You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gaborgsomogyi (via GitHub)" <gi...@apache.org> on 2025/02/13 09:09:31 UTC
[PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
gaborgsomogyi opened a new pull request, #26154:
URL: https://github.com/apache/flink/pull/26154
## What is the purpose of the change
[FLIP-496](https://cwiki.apache.org/confluence/display/FLINK/FLIP-496%3A+SQL+connector+for+keyed+savepoint+data) has described the SQL connector for keyed savepoint data feature in-depth. In this PR I've added documentation for this functionality.
## Brief change log
Add SQL connector documentation for keyed savepoint data.
## Verifying this change
Built docs locally.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): yes
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954521009
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
Review Comment:
nit: It's -> it's
to be inferred -> to infer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954514450
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
Review Comment:
nits:
Can't -> can't
UID hash -> operator.uid.hash
same for line below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954833554
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
Review Comment:
There is no 1to1 mapping between SQL->Java type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954846568
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1956024216
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,119 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before you interrogate state using the table API, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), also known as partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by querying a table created using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
Review Comment:
I thought we can just remove it, the normal users are not able to distinguish the possible values and supported ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954826270
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954808868
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954831185
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
+
+| Flink SQL type | Java type |
+|-------------------------|-------------------------------------------------------------------------|
+| CHAR / VARCHAR / STRING | java.lang.String |
+| BOOLEAN | boolean |
Review Comment:
boolean and Boolean can also be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954828793
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954824761
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
Review Comment:
This is a generic pattern across SQL connectors without deep explanation. See datagen...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954834087
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
+
+| Flink SQL type | Java type |
+|-------------------------|-------------------------------------------------------------------------|
+| CHAR / VARCHAR / STRING | java.lang.String |
+| BOOLEAN | boolean |
+| BINARY / VARBINARY | byte[] |
+| DECIMAL | org.apache.flink.table.data.DecimalData |
+| TINYINT | byte |
+| SMALLINT | short |
+| INTEGER | int |
+| BIGINT | long |
+| FLOAT | float |
+| DOUBLE | double |
+| DATE | int |
Review Comment:
When not listed then not supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954530982
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
+
+| Flink SQL type | Java type |
+|-------------------------|-------------------------------------------------------------------------|
+| CHAR / VARCHAR / STRING | java.lang.String |
+| BOOLEAN | boolean |
Review Comment:
I am curious if the content can be nullable - maybe Boolean would be more representative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954521009
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
Review Comment:
nit: It's -> it's
to be inferred -> to infer
same for the line below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954505598
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
Review Comment:
nit: the create table is not reading it. Maybe change this to "Then it can read by querying a table created using the following SQL statement"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #26154:
URL: https://github.com/apache/flink/pull/26154#issuecomment-2656303266
cc @gyfora @Zakelly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954500835
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
Review Comment:
nit: `or partitioned state` -> also known as `partitioned state`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954812673
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954518842
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
Review Comment:
nit : tries to be inferred -> tries to infer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #26154:
URL: https://github.com/apache/flink/pull/26154#issuecomment-2658682417
> And I suggest another ticket for the chinese doc.
Just created https://issues.apache.org/jira/browse/FLINK-37325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954511630
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
Review Comment:
nits:
filesystem -> file systems
which -> that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954828028
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954532918
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
+
+| Flink SQL type | Java type |
+|-------------------------|-------------------------------------------------------------------------|
+| CHAR / VARCHAR / STRING | java.lang.String |
+| BOOLEAN | boolean |
+| BINARY / VARBINARY | byte[] |
+| DECIMAL | org.apache.flink.table.data.DecimalData |
+| TINYINT | byte |
+| SMALLINT | short |
+| INTEGER | int |
+| BIGINT | long |
+| FLOAT | float |
+| DOUBLE | double |
+| DATE | int |
Review Comment:
what about date time and time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954537188
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -26,7 +26,7 @@ under the License.
# State Processor API
-Apache Flink's State Processor API provides powerful functionality to reading, writing, and modifying savepoints and checkpoints using Flink’s DataStream API under `BATCH` execution.
+Apache Flink's State Processor API provides powerful functionality to reading, writing, and modifying savepoints and checkpoints using Flink’s [DataStream API]({{< ref "#datastream-api" >}}) and [Table API]({{< ref "#table-api" >}}) under `BATCH` execution.
Review Comment:
nits:
powerful functionality to -> powerful functionality for
under -> using
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954811543
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
Review Comment:
Link to where? The whole doc has no ValueState chapter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954522469
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
Review Comment:
nit: is inferring -> infers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954815138
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954827349
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on PR #26154:
URL: https://github.com/apache/flink/pull/26154#issuecomment-2658230136
And I suggest another ticket for the chinese doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1955788750
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,119 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before you interrogate state using the table API, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), also known as partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by querying a table created using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
Review Comment:
Yeah, practically the code can eat the value but will blow up with exception. In short the possible values and support now here differs. If you think we can just remove it for now. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954521009
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
Review Comment:
nit: It's -> it's
to be inferred -> to infer
`map-key-format` is a strange config option name, as format is overloaded with connectors. I would suggest `map-key-type` and `value-type` for value.
same for the line below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #26154:
URL: https://github.com/apache/flink/pull/26154#issuecomment-2657188030
@davidradl thanks for the suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954535936
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
+
+| Flink SQL type | Java type |
+|-------------------------|-------------------------------------------------------------------------|
+| CHAR / VARCHAR / STRING | java.lang.String |
+| BOOLEAN | boolean |
+| BINARY / VARBINARY | byte[] |
+| DECIMAL | org.apache.flink.table.data.DecimalData |
+| TINYINT | byte |
+| SMALLINT | short |
+| INTEGER | int |
+| BIGINT | long |
+| FLOAT | float |
+| DOUBLE | double |
+| DATE | int |
+| INTERVAL_YEAR_MONTH | long |
+| INTERVAL_DAY_TIME | long |
+| ARRAY | java.util.List |
+| MAP | java.util.Map |
+| ROW | java.util.List\<org.apache.flink.table.types.logical.RowType.RowField\> |
+
+SHORTCUT: When a complex java class is defined in a column with STRING SQL type then the class instance
+`toString` method result will be the column value. This can be useful when only explanatory query needs to be
Review Comment:
nit:
This can be useful when only explanatory query needs to be developed in short time. ->
This can be useful when a quick explanatory query is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954537188
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -26,7 +26,7 @@ under the License.
# State Processor API
-Apache Flink's State Processor API provides powerful functionality to reading, writing, and modifying savepoints and checkpoints using Flink’s DataStream API under `BATCH` execution.
+Apache Flink's State Processor API provides powerful functionality to reading, writing, and modifying savepoints and checkpoints using Flink’s [DataStream API]({{< ref "#datastream-api" >}}) and [Table API]({{< ref "#table-api" >}}) under `BATCH` execution.
Review Comment:
nits:
powerful functionality to -> powerful functionality for
under -> using or executing in BATCH mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1955808914
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,119 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before you interrogate state using the table API, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), also known as partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by querying a table created using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All file system that are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (can't be used together with `operator.uid.hash`). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (can't be used together with `operator.uid`). |
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954499033
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
Review Comment:
nit: remove `is now`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954813972
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
Review Comment:
Sometimes yes, but since not all the time this config must exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1956062376
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,119 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before you interrogate state using the table API, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), also known as partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by querying a table created using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1955530219
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,119 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before you interrogate state using the table API, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), also known as partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by querying a table created using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All file system that are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (can't be used together with `operator.uid.hash`). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (can't be used together with `operator.uid`). |
Review Comment:
Additional note that either `operator.uid` or `operator.uid.hash` must be specified?
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,119 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before you interrogate state using the table API, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), also known as partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by querying a table created using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
Review Comment:
```suggestion
| connector | required | (none) | String | Specify what connector to use, here should be 'savepoint'. |
```
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,119 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before you interrogate state using the table API, make sure to review our [Flink SQL](../development/sql.md) guidelines.
Review Comment:
Seems invalid link?
```suggestion
Before you interrogate state using the table API, make sure to review our [Flink SQL](({{< ref "docs/dev/table/overview" >}})) guidelines.
```
OR
```suggestion
Before you interrogate state using the table API, make sure to review our [Flink SQL](({{< ref "docs/dev/table/sql/overview" >}})) guidelines.
```
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,119 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before you interrogate state using the table API, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), also known as partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by querying a table created using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
Review Comment:
IIUC, `forst` will not be supported if https://github.com/apache/flink/pull/26134/files#diff-c6d7f03b3d79d772c126e131a60d031f19a8fb3ac0a68d6cda81dfd2d216ff7cR388 is merged, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954545970
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
Review Comment:
we refer to "state SQL connector" before mentioning it . Maybe we could make this more general. Before you interrogate state using the table API, make sure to ....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954515580
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
Review Comment:
It would be worth describing what column ‘#’ means.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954501891
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
Review Comment:
nit: ValueState could be a link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954817814
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954834966
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
+
+| Flink SQL type | Java type |
+|-------------------------|-------------------------------------------------------------------------|
+| CHAR / VARCHAR / STRING | java.lang.String |
+| BOOLEAN | boolean |
+| BINARY / VARBINARY | byte[] |
+| DECIMAL | org.apache.flink.table.data.DecimalData |
+| TINYINT | byte |
+| SMALLINT | short |
+| INTEGER | int |
+| BIGINT | long |
+| FLOAT | float |
+| DOUBLE | double |
+| DATE | int |
+| INTERVAL_YEAR_MONTH | long |
+| INTERVAL_DAY_TIME | long |
+| ARRAY | java.util.List |
+| MAP | java.util.Map |
+| ROW | java.util.List\<org.apache.flink.table.types.logical.RowType.RowField\> |
+
+SHORTCUT: When a complex java class is defined in a column with STRING SQL type then the class instance
+`toString` method result will be the column value. This can be useful when only explanatory query needs to be
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954836186
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -26,7 +26,7 @@ under the License.
# State Processor API
-Apache Flink's State Processor API provides powerful functionality to reading, writing, and modifying savepoints and checkpoints using Flink’s DataStream API under `BATCH` execution.
+Apache Flink's State Processor API provides powerful functionality to reading, writing, and modifying savepoints and checkpoints using Flink’s [DataStream API]({{< ref "#datastream-api" >}}) and [Table API]({{< ref "#table-api" >}}) under `BATCH` execution.
Review Comment:
This is not added by me but let's make it better so changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954809951
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26154:
URL: https://github.com/apache/flink/pull/26154#issuecomment-2656020195
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bb3710c462ac08e3e2c4f37bba7f952258729b33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb3710c462ac08e3e2c4f37bba7f952258729b33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bb3710c462ac08e3e2c4f37bba7f952258729b33 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954529204
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
Review Comment:
nit: bases -> basis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954528887
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
Review Comment:
when can "If the mapping is not calculated properly" occur?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954510347
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
Review Comment:
Is state.backend.type required. Can we not determine this dynamically from the state format in the statepath?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954530982
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -514,3 +512,120 @@ savepointWriter
OperatorIdentifier.forUid("new-uid"))
...
```
+
+## Table API
+
+### Getting started
+
+Before getting started with state SQL connector, make sure to review our [Flink SQL](../development/sql.md) guidelines.
+
+IMPORTANT NOTE: State Table API is now only supports keyed state.
+
+### Keyed State
+
+[Keyed state]({{< ref "docs/dev/datastream/fault-tolerance/state" >}}#keyed-state), or partitioned state, is any state that is partitioned relative to a key.
+
+The SQL connector allows users to read arbitrary columns as ValueState and complex state types such as ListState, MapState.
+This means if an operator contains a stateful process function such as:
+```java
+eventStream
+ .keyBy(e -> (Integer)e.key)
+ .process(new StatefulFunction())
+ .uid("my-uid");
+
+...
+
+public class Account {
+ private Integer id;
+ public Double amount;
+
+ public Integer geId() {
+ return id;
+ }
+
+ public void setId(Integer id) {
+ this.id = id;
+ }
+}
+
+public class StatefulFunction extends KeyedProcessFunction<Integer, Integer, Void> {
+ private ValueState<Integer> myValueState;
+ private ValueState<Account> myAccountValueState;
+ private ListState<Integer> myListState;
+ private MapState<Integer, Integer> myMapState;
+
+ @Override
+ public void open(OpenContext openContext) {
+ myValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyValueState", Integer.class));
+ myAccountValueState = getRuntimeContext().getState(new ValueStateDescriptor<>("MyAccountValueState", Account.class));
+ myValueState = getRuntimeContext().getListState(new ListStateDescriptor<>("MyListState", Integer.class));
+ myMapState = getRuntimeContext().getMapState(new MapStateDescriptor<>("MyMapState", Integer.class, Integer.class));
+ }
+ ...
+}
+```
+
+Then it can read by using the following SQL statement:
+```SQL
+CREATE TABLE state_table (
+ k INTEGER,
+ MyValueState INTEGER,
+ MyAccountValueState ROW<id INTEGER, amount DOUBLE>,
+ MyListState ARRAY<INTEGER>,
+ MyMapState MAP<INTEGER, INTEGER>,
+ PRIMARY KEY (k) NOT ENFORCED
+) WITH (
+ 'connector' = 'savepoint',
+ 'state.backend.type' = 'rocksdb',
+ 'state.path' = '/root/dir/of/checkpoint-data/chk-1',
+ 'operator.uid' = 'my-uid'
+);
+```
+
+### Connector options
+
+#### General options
+| Option | Required | Default | Type | Description |
+|--------------------|----------|---------|-----------------------------------------------|-------------|
+| connector | required | (none) | String | Specify what connector to use, here should be 'state'. |
+| state.backend.type | required | (none) | Enum Possible values: hashmap, rocksdb, forst | Defines the state backend which must be used for state reading. This must match with the value which was defined in Flink job which created the savepoint or checkpoint. |
+| state.path | required | (none) | String | Defines the state path which must be used for state reading. All filesystem which are supported by Flink can be used here. |
+| operator.uid | optional | (none) | String | Defines the operator UID which must be used for state reading (Can't be used together with UID hash). |
+| operator.uid.hash | optional | (none) | String | Defines the operator UID hash which must be used for state reading (Can't be used together with UID). |
+
+#### Connector options for column ‘#’
+| Option | Required | Default | Type | Description |
+|-------------------------|----------|---------|--------|-------------|
+| fields.#.state-name | optional | (none) | String | Overrides the state name which must be used for state reading. This can be useful when the state name contains characters which are not compliant with SQL column names. |
+| fields.#.state-type | optional | (none) | Enum Possible values: list, map, value | Defines the state type which must be used for state reading, including value, list and map. When It's not provided then it tries to be inferred from the SQL type (ARRAY=list, MAP=map, all others=value). |
+| fields.#.map-key-format | optional | (none) | String | Defines the format class scheme for decoding map value key data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+| fields.#.value-format | optional | (none) | String | Defines the format class scheme for decoding value data (for ex. java.lang.Long). When It's not provided then it tries to be inferred from the SQL type (only primitive types supported). |
+
+### Default Data Type Mapping
+
+The state SQL connector is inferring the data type for primitive types when `fields.#.value-format` and `fields.#.map-key-format`
+are not defined. The following table shows the `Flink SQL type` -> `Java type` default mapping. If the mapping is not calculated properly
+then it can be overridden with the two mentioned config parameters on a per column bases.
+
+| Flink SQL type | Java type |
+|-------------------------|-------------------------------------------------------------------------|
+| CHAR / VARCHAR / STRING | java.lang.String |
+| BOOLEAN | boolean |
Review Comment:
I am curious if the content can be nullable - maybe Boolean would be a more representative Java type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26154:
URL: https://github.com/apache/flink/pull/26154#discussion_r1954537188
##########
docs/content/docs/libs/state_processor_api.md:
##########
@@ -26,7 +26,7 @@ under the License.
# State Processor API
-Apache Flink's State Processor API provides powerful functionality to reading, writing, and modifying savepoints and checkpoints using Flink’s DataStream API under `BATCH` execution.
+Apache Flink's State Processor API provides powerful functionality to reading, writing, and modifying savepoints and checkpoints using Flink’s [DataStream API]({{< ref "#datastream-api" >}}) and [Table API]({{< ref "#table-api" >}}) under `BATCH` execution.
Review Comment:
nit: powerful functionality to -> powerful functionality for
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37310][doc] SQL connector for keyed savepoint data documentation [flink]
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi merged PR #26154:
URL: https://github.com/apache/flink/pull/26154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org