[jira] [Created] (SAMZA-2383) Make Samza internal state (backed by RocksDB) queryable

["Dengpan Yin (Jira)" <ji...@apache.org>]

Dengpan Yin created SAMZA-2383:
----------------------------------

             Summary: Make Samza internal state (backed by RocksDB) queryable
                 Key: SAMZA-2383
                 URL: https://issues.apache.org/jira/browse/SAMZA-2383
             Project: Samza
          Issue Type: Task
          Components: container
            Reporter: Dengpan Yin
            Assignee: Dengpan Yin


Samza is using RocksDB as the KV store for internal state, both key and values are encoded as byte array.  Some application owners want to query the internal state when the job is running for troubleshooting.  To do so, the application owner needs to access the cluster where the container is running.  For security reasons, the application owner usually does not haver permission to access the shared cluster, which prevent the application owner from querying the RocksDB directly. 

Samza-beam-runner also include some meta information in the key, such as the window information.   The application owner usually does not know the window information when perform the query. The application owner needs to understand the encoding format in order to do a range query.

 

It will be helpful to provide a convenient interface to the user to query the state and show the results.  For instance, the user can specify the container ID, partition#, the state name, the query key and SERDE.  The request will be processed by the container and present the results to the user.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)