Re: [DISCUSS] FLIP-XXX: Enhancing HistoryServer with Lazy Loading and Pluggable Storage

[Feng Jin <ji...@gmail.com>]

Hi Zihao,

Thanks for initiating this FLIP. I think this proposal could be very
helpful for improving the performance and overall user experience of
accessing archived jobs in Flink.

From the current proposal, lazy loading and a pluggable local storage
backend seem like two relatively independent pieces of functionality. Would
it make sense to split them into two separate FLIPs?


Best,
Feng


On Mon, Mar 23, 2026 at 8:02 PM zihao chen <is...@gmail.com> wrote:

> Hi all,
>
> I’d like to start a discussion on *FLIP-XXX: Enhancing HistoryServer with
> Lazy Loading and Pluggable Storage*.
>
> This FLIP proposes improving the HistoryServer to address *slow
> startup*, *delayed
> job visibility*, and *excessive small files* when handling large numbers of
> archived jobs.
>
> The proposal introduces:
>
>    - *Lazy loading mode* for on-demand job archive fetching
>    - Optional *RocksDB-based storage* to reduce small files
>
> Compatibility: Full backward compatibility (EAGER + FILE as default)
>
> The detailed designs are described in the FLIP document:
>
>
> https://docs.google.com/document/d/1FQn-HQSIdhio3ImeOrbS9VfB6JqRTrRV4ZEZDTf3K2o/edit?usp=sharing
>
> Looking forward to your feedback.
>
>
> Best regards,
>
> Zihao Chen
>

Re: [DISCUSS] FLIP-XXX: Enhancing HistoryServer with Lazy Loading and Pluggable Storage

[zihao chen <is...@gmail.com>]

Hi Feng,

Thanks a lot for your suggestion!

I agree that lazy loading and pluggable storage can be treated as two
relatively independent pieces of functionality. Splitting them into
separate FLIPs makes sense and could improve clarity and reviewability.

Based on this, I’m planning to:

   - Extract *lazy loading* as one FLIP, focusing on on-demand archive
   loading and access patterns
   - Extract *pluggable storage* as another FLIP, focusing on the storage
   abstraction and backend implementations (e.g., RocksDB)

I will reorganize the current proposal accordingly and share the updated
versions soon.

Thanks again for the helpful feedback!

Best,
Zihao

Feng Jin <ji...@gmail.com> 于2026年4月14日周二 15:08写道：

> Hi Zihao,
>
> Thanks for initiating this FLIP. I think this proposal could be very
> helpful for improving the performance and overall user experience of
> accessing archived jobs in Flink.
>
> From the current proposal, lazy loading and a pluggable local storage
> backend seem like two relatively independent pieces of functionality. Would
> it make sense to split them into two separate FLIPs?
>
>
> Best,
> Feng
>
>
> On Mon, Mar 23, 2026 at 8:02 PM zihao chen <is...@gmail.com> wrote:
>
> > Hi all,
> >
> > I’d like to start a discussion on *FLIP-XXX: Enhancing HistoryServer with
> > Lazy Loading and Pluggable Storage*.
> >
> > This FLIP proposes improving the HistoryServer to address *slow
> > startup*, *delayed
> > job visibility*, and *excessive small files* when handling large numbers
> of
> > archived jobs.
> >
> > The proposal introduces:
> >
> >    - *Lazy loading mode* for on-demand job archive fetching
> >    - Optional *RocksDB-based storage* to reduce small files
> >
> > Compatibility: Full backward compatibility (EAGER + FILE as default)
> >
> > The detailed designs are described in the FLIP document:
> >
> >
> >
> https://docs.google.com/document/d/1FQn-HQSIdhio3ImeOrbS9VfB6JqRTrRV4ZEZDTf3K2o/edit?usp=sharing
> >
> > Looking forward to your feedback.
> >
> >
> > Best regards,
> >
> > Zihao Chen
> >
>