[jira] [Created] (FLINK-23989) Flink SQL visibility

["wangzhihao (Jira)" <ji...@apache.org>]

wangzhihao created FLINK-23989:
----------------------------------

             Summary: Flink SQL visibility 
                 Key: FLINK-23989
                 URL: https://issues.apache.org/jira/browse/FLINK-23989
             Project: Flink
          Issue Type: New Feature
          Components: Runtime / Queryable State
            Reporter: wangzhihao


It’s desired to inspect into the internal states generated by SQL especially for debug purpose. We propose to add a new feature in Flink UI console, when the users click on a Vertex of the Job DAG. We shall list all states the vertex has in a separate panel (let’s call *states panel*). On this panel users can query the states with some keys. The returned value share be a human readable string instead of opaque binary.

 

Particularly, we need expose the states as queryable. But currently the user experience of queryable states is cumbersome. Only the serialized value is returned to client and users need to handle deserialization by themselves. What’s worse, the client need to construct the serializer and type manually. To improve this situation. We propose:
 # Have a new API to find all queryable states associated to a job vertex. This can be done to check against the KvStateLocationRegistry, which store the mapping between JobVertexId and states.
 # Have a new API to allow users get the types of queryable states: For a register name (String), Queryable Server will return the type of key and value ([LogicalType|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/logical/LogicalType.java]).
 # To generate human readable string with API in step 2, we can 1) generate [TypeSerializer|https://github.com/apache/flink/blob/6c9818323b41a84137c52822d2993df788dbc9bb/flink-core/src/main/java/org/apache/flink/api/common/typeutils/TypeSerializer.java] from the LogicalType, so as to handle Serde automatically. 2) to convert internal data structures to external data structures to generate printable string. (with converters [DataStructureConverter|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/table/data/conversion/DataStructureConverter.html] )

With all these steps and some modifications to Web UI/ Rest, we can enable users to query SQL internal states.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)