[jira] [Commented] (FLINK-36429) Enhancing Flink History Server File Storage and Retrieval with RocksDB

["Xiaowen Sun (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-36429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17907674#comment-17907674 ] 

Xiaowen Sun commented on FLINK-36429:
-------------------------------------

Hi folks,

I’ve created a PR for addressing the ticket. This POC integrates RocksDB as a pluggable backend for the Flink History Server to improve scalability and performance.

You can find the details and review the changes here: [https://github.com/apache/flink/pull/25838]

Looking forward to your feedback!

> Enhancing Flink History Server File Storage and Retrieval with RocksDB
> ----------------------------------------------------------------------
>
>                 Key: FLINK-36429
>                 URL: https://issues.apache.org/jira/browse/FLINK-36429
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>    Affects Versions: 1.20.0
>            Reporter: Xiaowen Sun
>            Priority: Major
>              Labels: historyserver, pull-request-available
>   Original Estimate: 2,016h
>  Remaining Estimate: 2,016h
>
> Currently, when a Flink job finishes, it writes an archive as a single file that maps paths to JSON files. Flink History Server (FHS) job archives are pulled locally where the FHS is running on, and this process creates a local directory that expands based on the contents of the single archive file.
> Because of how the FHS stores the files, there are a large number of directories created in the local file system. This system can become inefficient and slow as the volume of job archives increases, creating bottlenecks in job data navigation and retrieval.
> To illustrate the problem of inode usage, let’s consider a scenario where there are 5000 subtasks. Each subtask creates its own directory, and within each subtask directory, there are additional directories that might store only a single file. This structure rapidly increases the number of inodes consumed.
> Integrating RocksDB, a high-performance embedded database for key-value data, aims to resolve these issues by offering faster data access and better scalability. This integration is expected to significantly enhance the operational efficiency of FHS by allowing faster data retrieval and enabling a larger cache on local Kubernetes deployments, thus overcoming inode limitations



--
This message was sent by Atlassian Jira
(v8.20.10#820010)