You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Yi Zhang <zh...@163.com> on 2026/05/27 06:11:21 UTC
[DISCUSS] FLIP-582: Support RpcOperator Service
Hi everyone,
I would like to start a discussion on FLIP-582: Support RpcOperator Service [1].
AI-oriented workloads like multimodal data processing and model inference are
growing rapidly in recent years. These workloads are characterized by expensive
resources (GPUs) and high initialization costs (seconds to minutes for model
loading). In today's Flink, embedding them in the data plane couples their
parallelism and failover with surrounding operators; deploying them as external
services disconnects their lifecycle from the job and doubles operational overhead.
This FLIP introduces RpcOperator Service — a framework-level primitive that runs
user-defined compute as RPC services in an independent Pipelined Region within
the Flink job. Because the service is isolated at the scheduling level, it can achieve
fault isolation, independent scaling, and dedicated resource allocation. As a native
Flink primitive, it also lays the foundation for automatic flow control, flexible load
balancing, and coordinated auto-scaling — all without introducing external
infrastructure or additional operational burden.
Looking forward to your feedback and suggestions!
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
Best Regards,
Yi Zhang
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Yuan,
Thank you for bringing up the remote compaction and shared state perspectives — these
are excellent examples of how RPC Operator can serve beyond AI workloads. That said,
making it fully usable in these scenarios still requires significant follow-up work. This FLIP
is the first step — establishing the core primitive and interfaces.
1. Remote Compaction & SQL integration
Remote compaction is a great use case. RPC Operator can naturally satisfy its core
requirements, but SQL integration is indeed a key question. We see two approaches for
using RPC Operator services from SQL:
a. Explicit usage
Users define an RPC service through extended SQL DDL and access it explicitly from
UDFs via `FunctionContext.getRosClient()`. This gives users full control over service
definition and invocation.
b. Implicit rewriting
Users annotate a SQL UDF to indicate that it should be decoupled from the data plane.
The engine automatically rewrites the UDF into an independent RPC Operator service
and replaces the original topology node with a lightweight proxy operator that forwards
requests to the service. This is precisely the "Automatic Rewriting" capability described
in the Follow-up Tasks section.
Both approaches build on top of the core interfaces proposed in this FLIP. However, each
involves substantial design work of its own — SQL DDL extensions, planner integration,
and rewriting rules — and deserves dedicated discussion in follow-up FLIPs.
2. Stateful service & checkpoint support
The current FLIP intentionally scopes RPC Operator as stateless, which is a pragmatic
trade-off between implementation complexity and scenario coverage. Looking ahead,
stateful service support is an important direction. Even without requiring global consistency
with the data plane, enabling an independent consistency model on the RPC Operator
side (e.g., independent snapshots with at-least-once or best-effort guarantees) would
significantly broaden the range of use cases. I think this is well worth a dedicated FLIP to
explore in depth.
Thank you again for the feedback! Please feel free to follow up if you have any further
questions or suggestions.
Best,
Yi
At 2026-06-03 17:53:32, "Yuan Mei" <yu...@gmail.com> wrote:
>Hi Yi and everyone,
>
>Thanks for the proposal! RPC Operator is a great extension to Flink and it
>can unlock a lot of Flink's capabilities, potentially, such as doing remote
>compaction for disaggregation and providing Flink with global/shareable
>state capabilities (the same state data can be accessed across parallelisms
>and across operators).
>
>However, I have a few questions and thoughts regarding the design:
>
> -
>
> Remote Compaction can be reused under the current RPC service setting.
> The current Remote Compaction service (PoC), due to HA and other
> considerations, *is set up as another Flink job.* The basic requirements
> for this service are:
> -
>
> Multiple operators and instances can access the same RPC service.
> -
>
> Configurable with different types of CUs and needs a basic load
> balancing policy.
>
> But a major issue here is that disaggregation currently
>mainly integrate with SQL operators. How & what is the plan to integrate
>RPC Service with SQL? Are there any potential issues (I
>didn't see any discussion regarding SQL integration in this FLIP)
>
> -
>
> From the proposal, the RpcOperator appears to be stateless and does not
> support checkpoints; it specifically mentioned the difference with* data
> plane*. If we want to support sharable state, checkpoint mechanisms
> might also be needed under different semantic consistency guarantees. Of
> course, this part can be discussed separately and does not need to block
> this FLIP.
>
>
>Best
>Yuan
>
>On Wed, Jun 3, 2026 at 2:21 PM Lei Yang <le...@gmail.com> wrote:
>
>> Hi Yi,
>>
>> Thank you for the great work on FLIP-582. As GPU resources become
>> increasingly valuable, RpcOperator can significantly improve resource
>> utilization and strengthening Flink’s competitiveness for AI workloads.I am
>> particularly interested in the failover and flexible scaling aspects of
>> this
>> FLIP, and I have two questions that I hope you can help clarify.
>>
>> 1. Relationship between RpcOperator and JobGraph recovery semantics
>>
>> The current FLIP models RpcOperator as an independent JobVertex in
>> the JobGraph. This means that although it can be isolated from the data
>> plane in terms of resources and regions, it still belongs to the same job
>> graph in terms of scheduling, recovery, and rescaling semantics.
>>
>> For example, in AdaptiveScheduler, rescaling or failure recovery may enter
>> the Restarting -> CreatingExecutionGraph path and rebuild the
>> ExecutionGraph.
>> An external autoscaler can also adjust the parallelism requirements of job
>> vertices through the JobResourceRequirements REST API; with
>> AdaptiveScheduler,
>> this may further trigger rescale / restart. In such cases, RpcOperator may
>> also
>> be restarted together with the data-plane tasks, since it is part of the
>> same
>> JobGraph / ExecutionGraph.
>>
>> This seems to leave some gap with the goal of RpcOperator being an
>> independent
>> service that is not affected by the data plane. Therefore, I would like to
>> confirm
>> whether the FLIP plans to introduce a more fine-grained recovery and
>> restart
>> mechanism, so that RpcOperator can restart, fail over, or rescale
>> independently
>> from data processing vertices.
>>
>> 2. Client support for future flexible scaling
>>
>> The FLIP mentions that RpcOperator instances are independent service
>> instances, and that an instance going online or offline should not affect
>> other
>> instances or the data processing flow. I understand that the current FLIP
>> may
>> not need to fully support dynamic scaling in the first phase. However, if
>> flexible
>> scaling of RpcOperator is expected in the future, the client may need to be
>> aware of changes in RpcOperator parallelism and the instance list.
>>
>> For example, during a future scale-out, the system may start a new
>> RpcOperator
>> instance without restarting existing ones. After the new instance becomes
>> ready,
>> the client needs to discover it in time and include it in request routing.
>> During scale-in,
>> the client also needs to detect instance removal in time and avoid sending
>> new
>> requests to instances that are about to exit.
>>
>> Therefore, I would like to confirm whether the current ROSClient design can
>> support push-based discovery of RpcOperator instance additions and removals
>> for future no-restart dynamic scaling.
>>
>> Best,
>> Lei
>>
>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>>
>> > Hi everyone,
>> >
>> >
>> >
>> > I would like to start a discussion on FLIP-582: Support RpcOperator
>> > Service [1].
>> >
>> >
>> > AI-oriented workloads like multimodal data processing and model inference
>> > are
>> > growing rapidly in recent years. These workloads are characterized by
>> > expensive
>> > resources (GPUs) and high initialization costs (seconds to minutes for
>> > model
>> > loading). In today's Flink, embedding them in the data plane couples
>> their
>> > parallelism and failover with surrounding operators; deploying them as
>> > external
>> > services disconnects their lifecycle from the job and doubles operational
>> > overhead.
>> >
>> >
>> > This FLIP introduces RpcOperator Service — a framework-level primitive
>> > that runs
>> > user-defined compute as RPC services in an independent Pipelined Region
>> > within
>> > the Flink job. Because the service is isolated at the scheduling level,
>> it
>> > can achieve
>> > fault isolation, independent scaling, and dedicated resource allocation.
>> > As a native
>> > Flink primitive, it also lays the foundation for automatic flow control,
>> > flexible load
>> > balancing, and coordinated auto-scaling — all without introducing
>> external
>> > infrastructure or additional operational burden.
>> >
>> >
>> >
>> >
>> > Looking forward to your feedback and suggestions!
>> >
>> >
>> >
>> >
>> > [1]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>> >
>> >
>> >
>> >
>> >
>> > Best Regards,
>> > Yi Zhang
>>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yuan Mei <yu...@gmail.com>.
Hi Yi and everyone,
Thanks for the proposal! RPC Operator is a great extension to Flink and it
can unlock a lot of Flink's capabilities, potentially, such as doing remote
compaction for disaggregation and providing Flink with global/shareable
state capabilities (the same state data can be accessed across parallelisms
and across operators).
However, I have a few questions and thoughts regarding the design:
-
Remote Compaction can be reused under the current RPC service setting.
The current Remote Compaction service (PoC), due to HA and other
considerations, *is set up as another Flink job.* The basic requirements
for this service are:
-
Multiple operators and instances can access the same RPC service.
-
Configurable with different types of CUs and needs a basic load
balancing policy.
But a major issue here is that disaggregation currently
mainly integrate with SQL operators. How & what is the plan to integrate
RPC Service with SQL? Are there any potential issues (I
didn't see any discussion regarding SQL integration in this FLIP)
-
From the proposal, the RpcOperator appears to be stateless and does not
support checkpoints; it specifically mentioned the difference with* data
plane*. If we want to support sharable state, checkpoint mechanisms
might also be needed under different semantic consistency guarantees. Of
course, this part can be discussed separately and does not need to block
this FLIP.
Best
Yuan
On Wed, Jun 3, 2026 at 2:21 PM Lei Yang <le...@gmail.com> wrote:
> Hi Yi,
>
> Thank you for the great work on FLIP-582. As GPU resources become
> increasingly valuable, RpcOperator can significantly improve resource
> utilization and strengthening Flink’s competitiveness for AI workloads.I am
> particularly interested in the failover and flexible scaling aspects of
> this
> FLIP, and I have two questions that I hope you can help clarify.
>
> 1. Relationship between RpcOperator and JobGraph recovery semantics
>
> The current FLIP models RpcOperator as an independent JobVertex in
> the JobGraph. This means that although it can be isolated from the data
> plane in terms of resources and regions, it still belongs to the same job
> graph in terms of scheduling, recovery, and rescaling semantics.
>
> For example, in AdaptiveScheduler, rescaling or failure recovery may enter
> the Restarting -> CreatingExecutionGraph path and rebuild the
> ExecutionGraph.
> An external autoscaler can also adjust the parallelism requirements of job
> vertices through the JobResourceRequirements REST API; with
> AdaptiveScheduler,
> this may further trigger rescale / restart. In such cases, RpcOperator may
> also
> be restarted together with the data-plane tasks, since it is part of the
> same
> JobGraph / ExecutionGraph.
>
> This seems to leave some gap with the goal of RpcOperator being an
> independent
> service that is not affected by the data plane. Therefore, I would like to
> confirm
> whether the FLIP plans to introduce a more fine-grained recovery and
> restart
> mechanism, so that RpcOperator can restart, fail over, or rescale
> independently
> from data processing vertices.
>
> 2. Client support for future flexible scaling
>
> The FLIP mentions that RpcOperator instances are independent service
> instances, and that an instance going online or offline should not affect
> other
> instances or the data processing flow. I understand that the current FLIP
> may
> not need to fully support dynamic scaling in the first phase. However, if
> flexible
> scaling of RpcOperator is expected in the future, the client may need to be
> aware of changes in RpcOperator parallelism and the instance list.
>
> For example, during a future scale-out, the system may start a new
> RpcOperator
> instance without restarting existing ones. After the new instance becomes
> ready,
> the client needs to discover it in time and include it in request routing.
> During scale-in,
> the client also needs to detect instance removal in time and avoid sending
> new
> requests to instances that are about to exit.
>
> Therefore, I would like to confirm whether the current ROSClient design can
> support push-based discovery of RpcOperator instance additions and removals
> for future no-restart dynamic scaling.
>
> Best,
> Lei
>
> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>
> > Hi everyone,
> >
> >
> >
> > I would like to start a discussion on FLIP-582: Support RpcOperator
> > Service [1].
> >
> >
> > AI-oriented workloads like multimodal data processing and model inference
> > are
> > growing rapidly in recent years. These workloads are characterized by
> > expensive
> > resources (GPUs) and high initialization costs (seconds to minutes for
> > model
> > loading). In today's Flink, embedding them in the data plane couples
> their
> > parallelism and failover with surrounding operators; deploying them as
> > external
> > services disconnects their lifecycle from the job and doubles operational
> > overhead.
> >
> >
> > This FLIP introduces RpcOperator Service — a framework-level primitive
> > that runs
> > user-defined compute as RPC services in an independent Pipelined Region
> > within
> > the Flink job. Because the service is isolated at the scheduling level,
> it
> > can achieve
> > fault isolation, independent scaling, and dedicated resource allocation.
> > As a native
> > Flink primitive, it also lays the foundation for automatic flow control,
> > flexible load
> > balancing, and coordinated auto-scaling — all without introducing
> external
> > infrastructure or additional operational burden.
> >
> >
> >
> >
> > Looking forward to your feedback and suggestions!
> >
> >
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
> >
> >
> >
> >
> >
> > Best Regards,
> > Yi Zhang
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Lei,
Thank you for the thorough review and the insightful feedback! Your questions on
failover and scaling are very valuable. Here are my responses:
1. Relationship between RpcOperator and JobGraph recovery semantics
This is a great observation. RpcOperator achieves failover isolation through an
independent Pipelined Region. Under the **DefaultScheduler**, this works as
intended — a subtask failure only restarts the affected Region, and the RpcOperator
Region remains unaffected (and vice versa).
However, the **AdaptiveScheduler** currently only supports full-graph restarts — it
cannot restart individual Regions independently. This means that during both failover
and rescale, the data plane and the service plane cannot be isolated: the entire
ExecutionGraph (including RpcOperator vertices) is torn down and recreated. This
is a limitation of the AdaptiveScheduler. Achieving true isolation under
AdaptiveScheduler would require enhancing it with region-level restart capabilities,
which is orthogonal to the RpcOperator design and should be pursued independently.
I will update the FLIP to clarify the scope of failover isolation with respect to different
schedulers.
2. Client support for future flexible scaling
Great point, and I fully agree. Supporting push-based endpoint updates is essential
for the client to discover instance additions and removals in a timely manner,
especially for future no-restart dynamic scaling. I will update the FLIP to incorporate
a push-based endpoint update mechanism in the service discovery design.
Thank you for the feedback and suggestions. Please feel free to follow up if you
have any further questions!
Best,
Yi
At 2026-06-03 14:20:00, "Lei Yang" <le...@gmail.com> wrote:
>Hi Yi,
>
>Thank you for the great work on FLIP-582. As GPU resources become
>increasingly valuable, RpcOperator can significantly improve resource
>utilization and strengthening Flink’s competitiveness for AI workloads.I am
>particularly interested in the failover and flexible scaling aspects of
>this
>FLIP, and I have two questions that I hope you can help clarify.
>
>1. Relationship between RpcOperator and JobGraph recovery semantics
>
>The current FLIP models RpcOperator as an independent JobVertex in
>the JobGraph. This means that although it can be isolated from the data
>plane in terms of resources and regions, it still belongs to the same job
>graph in terms of scheduling, recovery, and rescaling semantics.
>
>For example, in AdaptiveScheduler, rescaling or failure recovery may enter
>the Restarting -> CreatingExecutionGraph path and rebuild the
>ExecutionGraph.
>An external autoscaler can also adjust the parallelism requirements of job
>vertices through the JobResourceRequirements REST API; with
>AdaptiveScheduler,
>this may further trigger rescale / restart. In such cases, RpcOperator may
>also
>be restarted together with the data-plane tasks, since it is part of the
>same
>JobGraph / ExecutionGraph.
>
>This seems to leave some gap with the goal of RpcOperator being an
>independent
>service that is not affected by the data plane. Therefore, I would like to
>confirm
>whether the FLIP plans to introduce a more fine-grained recovery and
>restart
>mechanism, so that RpcOperator can restart, fail over, or rescale
>independently
>from data processing vertices.
>
>2. Client support for future flexible scaling
>
>The FLIP mentions that RpcOperator instances are independent service
>instances, and that an instance going online or offline should not affect
>other
>instances or the data processing flow. I understand that the current FLIP
>may
>not need to fully support dynamic scaling in the first phase. However, if
>flexible
>scaling of RpcOperator is expected in the future, the client may need to be
>aware of changes in RpcOperator parallelism and the instance list.
>
>For example, during a future scale-out, the system may start a new
>RpcOperator
> instance without restarting existing ones. After the new instance becomes
>ready,
>the client needs to discover it in time and include it in request routing.
>During scale-in,
>the client also needs to detect instance removal in time and avoid sending
>new
>requests to instances that are about to exit.
>
>Therefore, I would like to confirm whether the current ROSClient design can
>support push-based discovery of RpcOperator instance additions and removals
> for future no-restart dynamic scaling.
>
>Best,
>Lei
>
>Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>
>> Hi everyone,
>>
>>
>>
>> I would like to start a discussion on FLIP-582: Support RpcOperator
>> Service [1].
>>
>>
>> AI-oriented workloads like multimodal data processing and model inference
>> are
>> growing rapidly in recent years. These workloads are characterized by
>> expensive
>> resources (GPUs) and high initialization costs (seconds to minutes for
>> model
>> loading). In today's Flink, embedding them in the data plane couples their
>> parallelism and failover with surrounding operators; deploying them as
>> external
>> services disconnects their lifecycle from the job and doubles operational
>> overhead.
>>
>>
>> This FLIP introduces RpcOperator Service — a framework-level primitive
>> that runs
>> user-defined compute as RPC services in an independent Pipelined Region
>> within
>> the Flink job. Because the service is isolated at the scheduling level, it
>> can achieve
>> fault isolation, independent scaling, and dedicated resource allocation.
>> As a native
>> Flink primitive, it also lays the foundation for automatic flow control,
>> flexible load
>> balancing, and coordinated auto-scaling — all without introducing external
>> infrastructure or additional operational burden.
>>
>>
>>
>>
>> Looking forward to your feedback and suggestions!
>>
>>
>>
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>>
>>
>>
>>
>>
>> Best Regards,
>> Yi Zhang
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Lei Yang <le...@gmail.com>.
Hi Yi,
Thank you for the great work on FLIP-582. As GPU resources become
increasingly valuable, RpcOperator can significantly improve resource
utilization and strengthening Flink’s competitiveness for AI workloads.I am
particularly interested in the failover and flexible scaling aspects of
this
FLIP, and I have two questions that I hope you can help clarify.
1. Relationship between RpcOperator and JobGraph recovery semantics
The current FLIP models RpcOperator as an independent JobVertex in
the JobGraph. This means that although it can be isolated from the data
plane in terms of resources and regions, it still belongs to the same job
graph in terms of scheduling, recovery, and rescaling semantics.
For example, in AdaptiveScheduler, rescaling or failure recovery may enter
the Restarting -> CreatingExecutionGraph path and rebuild the
ExecutionGraph.
An external autoscaler can also adjust the parallelism requirements of job
vertices through the JobResourceRequirements REST API; with
AdaptiveScheduler,
this may further trigger rescale / restart. In such cases, RpcOperator may
also
be restarted together with the data-plane tasks, since it is part of the
same
JobGraph / ExecutionGraph.
This seems to leave some gap with the goal of RpcOperator being an
independent
service that is not affected by the data plane. Therefore, I would like to
confirm
whether the FLIP plans to introduce a more fine-grained recovery and
restart
mechanism, so that RpcOperator can restart, fail over, or rescale
independently
from data processing vertices.
2. Client support for future flexible scaling
The FLIP mentions that RpcOperator instances are independent service
instances, and that an instance going online or offline should not affect
other
instances or the data processing flow. I understand that the current FLIP
may
not need to fully support dynamic scaling in the first phase. However, if
flexible
scaling of RpcOperator is expected in the future, the client may need to be
aware of changes in RpcOperator parallelism and the instance list.
For example, during a future scale-out, the system may start a new
RpcOperator
instance without restarting existing ones. After the new instance becomes
ready,
the client needs to discover it in time and include it in request routing.
During scale-in,
the client also needs to detect instance removal in time and avoid sending
new
requests to instances that are about to exit.
Therefore, I would like to confirm whether the current ROSClient design can
support push-based discovery of RpcOperator instance additions and removals
for future no-restart dynamic scaling.
Best,
Lei
Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
> Hi everyone,
>
>
>
> I would like to start a discussion on FLIP-582: Support RpcOperator
> Service [1].
>
>
> AI-oriented workloads like multimodal data processing and model inference
> are
> growing rapidly in recent years. These workloads are characterized by
> expensive
> resources (GPUs) and high initialization costs (seconds to minutes for
> model
> loading). In today's Flink, embedding them in the data plane couples their
> parallelism and failover with surrounding operators; deploying them as
> external
> services disconnects their lifecycle from the job and doubles operational
> overhead.
>
>
> This FLIP introduces RpcOperator Service — a framework-level primitive
> that runs
> user-defined compute as RPC services in an independent Pipelined Region
> within
> the Flink job. Because the service is isolated at the scheduling level, it
> can achieve
> fault isolation, independent scaling, and dedicated resource allocation.
> As a native
> Flink primitive, it also lays the foundation for automatic flow control,
> flexible load
> balancing, and coordinated auto-scaling — all without introducing external
> infrastructure or additional operational burden.
>
>
>
>
> Looking forward to your feedback and suggestions!
>
>
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>
>
>
>
>
> Best Regards,
> Yi Zhang
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Xintong,
Thank you for the support and for the thoughtful analysis! Your perspective from both the stateful
stream processing model and the Flink Agents side is very valuable. The subagent pattern you
described — bursty workloads, dynamic resource demands, and the need for a shared, elastically
scaling resource pool — maps precisely onto what RPC Operator provides. This reinforces our
belief that RPC Operator is a general-purpose deployment primitive. And the future auto-scaling
support will further enhance its ability to handle such bursty workloads.
Regarding the communication layer and large message sizes — this is a practical concern
that deserves attention. Flink already provides a configuration option `pekko.framesize` that
controls the maximum size of a single RPC message, which can be increased to accommodate
larger payloads and should be sufficient for most text-based inference scenarios. For workloads
involving very large payloads, passing data references rather than the raw data in the RPC
message is a more practical approach. Looking further ahead, the communication layer is
designed to be swappable: transport implementations such as gRPC or Netty can be introduced
without affecting the upper-level interfaces, and these alternatives natively support streaming
and chunked transfer for large-payload scenarios. I will add a note in the FLIP to clarify the
message size considerations and recommended approaches.
Thanks again for the feedback! Please feel free to follow up if you have any further questions or
suggestions.
Best,
Yi
At 2026-05-31 22:53:33, "Xintong Song" <to...@gmail.com> wrote:
>Hi Yi,
>
>Thanks for the FLIP. +1 from my side.
>
>Requiring operators that are connected and communicating to restart and
>rescale as a whole is a necessary constraint in Flink's stateful stream
>processing model, since that's what keeps state consistent. But in AI
>scenarios, many of the heavy, costly operators (model inference, multimodal
>processing, and the like) are themselves stateless. Their real problem is
>high initialization cost and expensive resources, and applying the same
>all-or-nothing constraint to them turns into a serious burden. This FLIP
>decouples exactly these operators from the data plane's Pipelined Region,
>which addresses the mismatch at its root. I like the direction.
>
>There's also a clear need for an rpc operator from the Flink Agents side.
>Take the subagent pattern: a main agent spawns one or more subagents to
>handle subtasks, and that workload is highly bursty. Within the same job,
>one moment's input may need several subagents while the next needs none.
>Statically allocating subagent resources per operator parallelism doesn't
>work well here: under-allocate and the operator gets blown out as soon as
>demand arrives, over-allocate and the resources sit idle most of the time.
>The natural fit is to put subagent execution into a shared resource pool
>that multiple upstream subtasks can use and that scales elastically with
>load, which maps directly onto what rpc operator provides: each instance
>forming its own Pipelined Region, giving independent scaling and flexible
>load balancing.
>
>One question on the communication layer. The first version reuses Pekko
>RPC, but the proposal only discusses latency overhead and QPS, not the size
>of a single message. Pekko has a default cap on per-message size, and in AI
>scenarios the context carried in a single call can be quite large
>(multimodal inputs, long context, etc.). Has this case been considered? How
>should large messages be handled, by chunking, by raising the limit, or by
>going through a different transport?
>
>Best,
>
>Xintong
>
>
>
>On Thu, May 28, 2026 at 7:35 PM Hongshun Wang <lo...@gmail.com>
>wrote:
>
>> Hi Yi,
>>
>> Thanks for the great work on FLIP-582. I've been reading through the
>> proposal and have a few questions regarding the failover/recovery semantics
>> and resource efficiency of RpcOperator Service. I'd appreciate any
>> clarification.
>>
>>
>> 1. Cost of recomputation due to stateless design
>>
>> The proposal states that "RPC services do not participate in
>> checkpointing." This means that upon any data-plane failure and checkpoint
>> rollback, all in-flight RPC requests are lost and must be re-triggered from
>> scratch. For workloads where a single RPC invocation is expensive (e.g.,
>> LLM inference taking seconds to minutes per request), this could lead to
>> significant wasted computation.
>>
>> Has there been any consideration of a lightweight mechanism to avoid
>> redundant recomputation — for example, a result cache on the service side
>> keyed by request identity, or an optional "resumable request" protocol? Or
>> is the position that this cost is acceptable given the target workload
>> characteristics?
>>
>>
>> 2. Cancelling in-flight RPC requests on data-plane failure
>>
>> When a data-plane operator fails and its Pipelined Region restarts, there
>> may still be RPC requests in progress on the service side that were issued
>> by the now-restarting data-plane tasks. Since those results will be
>> discarded anyway (the data plane will re-issue them after recovery),
>> continuing to process them wastes resources — particularly expensive GPU
>> cycles.
>>
>> Is there a plan to provide an interface or mechanism for the data plane (or
>> the framework) to proactively cancel outstanding RPC requests upon
>> failure/restart? For example, a cancellation signal propagated from the
>> RosClient to the RPC service, or a request-scoping mechanism tied to the
>> checkpoint epoch, so that the service side can abort stale requests early.
>>
>>
>> 3. Data-plane behavior when an RPC service subtask fails
>>
>> The proposal mentions that when an RPC service subtask fails, "requests are
>> routed to other surviving instances." However, I'm curious about the
>> behavior for requests that are already in-flight to the failed instance:
>>
>> - Does the data-plane operator have to wait for a timeout (as
>> configured) before it knows the request has failed?
>> - Is there any fast-failure notification mechanism (e.g.,
>> connection-closed detection) that allows the RosClient to immediately
>> fail
>> over in-flight requests to other instances without waiting for the full
>> timeout?
>> - If the service has only one instance (parallelism = 1) and it fails,
>> does the data plane simply block (with backpressure) until the instance
>> recovers, or does it fail fast with UNAVAILABLE errors?
>>
>> A fast-failure path would be important for reducing end-to-end latency
>> impact during service-side failures.
>>
>>
>> 4. Resource utilization concerns with separate SlotSharingGroup
>>
>> The proposal states that "each RPC service automatically creates a separate
>> SlotSharingGroup" and that this is "fully compatible with Flink's existing
>> K8s/YARN resource management." However, in Flink's current resource model,
>> all TaskManagers share the same configuration template (determined by
>> global taskmanager.memory.* and CPU settings). Fine-Grained Resource
>> Management controls how slots are sized within a TM, but does not enable
>> heterogeneous TM specifications across the cluster.
>>
>> This raises a practical concern: if an RpcOperator requires GPU resources,
>> the TM configuration must include GPU allocation. Since all TMs share the
>> same template, this means every TM in the cluster would be provisioned with
>> GPUs — even those exclusively hosting data-plane tasks that have no GPU
>> requirement. This leads to significant GPU resource waste.
>>
>> Furthermore, the separate SlotSharingGroup design prevents co-locating
>> RpcOperator and data-plane tasks on the same TaskManager. In practice,
>> co-location (e.g., RpcOperator using the GPU while data-plane tasks use the
>> CPU on the same GPU node) would improve overall resource utilization. The
>> current design prioritizes isolation at the cost of efficiency.
>>
>> Has there been consideration of:
>>
>> - Supporting heterogeneous TaskManager specifications (different TM
>> templates for different SlotSharingGroups) to avoid provisioning GPU on
>> CPU-only TMs?(As a reference, KubeRay's Worker Group model addresses
>> exactly this problem for Ray clusters. In KubeRay, a RayCluster defines
>> multiple workerGroupSpecs, each with its own pod template, resource
>> configuration, node selectors, and independent autoscaling bounds)
>> - Allowing optional co-location of RpcOperator and data-plane slots on
>> the same TM (in separate slots) for scenarios where utilization is
>> prioritized over strict process-level isolation?
>> - Or is there a dependency on other sub-FLIPs under FLIP-577 to address
>> this gap in GPU resource scheduling?
>>
>> Thanks in advance for the clarification. Looking forward to the continued
>> discussion.
>>
>>
>> Best regards
>>
>> Hongshun
>>
>> On Thu, May 28, 2026 at 4:40 PM Guowei Ma <gu...@gmail.com> wrote:
>>
>> > HI,YI
>> >
>> > One more question I'd like to raise is around how the service behaves
>> when
>> > it's referenced by more than one job. If two jobs are submitted within
>> the
>> > same application and both reference the same RpcOperator service, is that
>> > service shared across them, and what is the expected behavior in that
>> case?
>> > I'm also curious how this plays out for bounded (batch) jobs in
>> particular:
>> > if one job completes while the other is still running, what does the
>> > service lifecycle look like — is the service torn down as soon as the
>> first
>> > job finishes, and if so, how does that affect the job that's still
>> running?
>> >
>> > Best,
>> > Guowei
>> >
>> >
>> > On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
>> >
>> > > Hi,YI
>> > >
>> > > Thanks for putting together this FLIP — it's a well-thought-out
>> proposal,
>> > > and the RpcOperator Service is a really valuable addition for
>> > AI/inference
>> > > workloads on Flink. I have a few questions and suggestions:
>> > >
>> > >
>> > > 1. Autoscaling
>> > > How is scale-up/scale-down expressed in this design? Could you consider
>> > > allowing concurrency to be configured as a (min, max) range? Currently
>> it
>> > > appears only min = max (i.e. fixed parallelism) is supported.
>> > >
>> > >
>> > > 2. Async invocation & end-to-end examples
>> > > The example shows a synchronous invocation style. I'd suggest also
>> > > providing an asynchronous invocation example — or even just a scaffold
>> > that
>> > > exposes only the RpcOperator's handle and method. In addition, it would
>> > be
>> > > best to include a complete end-to-end example of invoking a model (CPU
>> > > inference would be fine).
>> > >
>> > >
>> > > 3. GPU as a resource
>> > > GPU is surely a common/first-class resource. Accessing it here through
>> an
>> > > extend mechanism — wouldn't that mislead users? It would seem more
>> > > natural to declare GPU as a standard resource rather than going through
>> > an
>> > > extension.
>> > >
>> > >
>> > > 4. Serial method invocation
>> > > Each method of an RpcOperator is invoked serially. I'd suggest stating
>> > > this explicitly in both the documentation and the code comments, so
>> that
>> > > users — or an AI — can be clearly aware of it.
>> > >
>> > >
>> > > 5. Why synchronous methods?
>> > > Why is every method in RpcOperator a synchronous call? Does this imply
>> > > that no model inference supports concurrent invocation?
>> > >
>> > >
>> > > 6. SQL UDF usage
>> > > How would SQL UDF users invoke this capability? I'd suggest providing a
>> > > complete example for this as well.
>> > >
>> > >
>> > > Thanks again for the great work!
>> > >
>> > > Best,
>> > > Guowei
>> > >
>> > >
>> > > On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
>> > > wrote:
>> > >
>> > >> Hi Yi, and Flink Community,
>> > >>
>> > >> Thanks for bringing up this excellent proposal. I am fully in favor of
>> > >> FLIP-582.
>> > >>
>> > >> In our production workloads, especially regarding large-scale AI
>> > >> inference,
>> > >> the tight coupling of the data plane and compute units has always
>> been a
>> > >> major pain point. If an inference subtask fails due to a GPU OOM or
>> > driver
>> > >> issue, triggering a global rollback is incredibly expensive since
>> model
>> > >> reloading takes minutes.
>> > >>
>> > >> The introduction of the RpcOperator Service as a first-class primitive
>> > >> masterfully decouples the heavy inference tasks from the mainstream
>> > >> topology. The fault isolation, independent scaling, and stateless
>> design
>> > >> perfectly match the requirements of modern AI-oriented data
>> processing.
>> > >>
>> > >> This is a clean and robust architecture. Looking forward to seeing
>> this
>> > >> merged!
>> > >>
>> > >>
>> > >> Best wishes,
>> > >> Charles Zhang
>> > >> from Apache InLong
>> > >>
>> > >>
>> > >> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>> > >>
>> > >> > Hi everyone,
>> > >> >
>> > >> >
>> > >> >
>> > >> > I would like to start a discussion on FLIP-582: Support RpcOperator
>> > >> > Service [1].
>> > >> >
>> > >> >
>> > >> > AI-oriented workloads like multimodal data processing and model
>> > >> inference
>> > >> > are
>> > >> > growing rapidly in recent years. These workloads are characterized
>> by
>> > >> > expensive
>> > >> > resources (GPUs) and high initialization costs (seconds to minutes
>> for
>> > >> > model
>> > >> > loading). In today's Flink, embedding them in the data plane couples
>> > >> their
>> > >> > parallelism and failover with surrounding operators; deploying them
>> as
>> > >> > external
>> > >> > services disconnects their lifecycle from the job and doubles
>> > >> operational
>> > >> > overhead.
>> > >> >
>> > >> >
>> > >> > This FLIP introduces RpcOperator Service — a framework-level
>> primitive
>> > >> > that runs
>> > >> > user-defined compute as RPC services in an independent Pipelined
>> > Region
>> > >> > within
>> > >> > the Flink job. Because the service is isolated at the scheduling
>> > level,
>> > >> it
>> > >> > can achieve
>> > >> > fault isolation, independent scaling, and dedicated resource
>> > allocation.
>> > >> > As a native
>> > >> > Flink primitive, it also lays the foundation for automatic flow
>> > control,
>> > >> > flexible load
>> > >> > balancing, and coordinated auto-scaling — all without introducing
>> > >> external
>> > >> > infrastructure or additional operational burden.
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > Looking forward to your feedback and suggestions!
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > [1]
>> > >> >
>> > >> >
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > Best Regards,
>> > >> > Yi Zhang
>> > >>
>> > >
>> >
>>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Xintong Song <to...@gmail.com>.
Hi Yi,
Thanks for the FLIP. +1 from my side.
Requiring operators that are connected and communicating to restart and
rescale as a whole is a necessary constraint in Flink's stateful stream
processing model, since that's what keeps state consistent. But in AI
scenarios, many of the heavy, costly operators (model inference, multimodal
processing, and the like) are themselves stateless. Their real problem is
high initialization cost and expensive resources, and applying the same
all-or-nothing constraint to them turns into a serious burden. This FLIP
decouples exactly these operators from the data plane's Pipelined Region,
which addresses the mismatch at its root. I like the direction.
There's also a clear need for an rpc operator from the Flink Agents side.
Take the subagent pattern: a main agent spawns one or more subagents to
handle subtasks, and that workload is highly bursty. Within the same job,
one moment's input may need several subagents while the next needs none.
Statically allocating subagent resources per operator parallelism doesn't
work well here: under-allocate and the operator gets blown out as soon as
demand arrives, over-allocate and the resources sit idle most of the time.
The natural fit is to put subagent execution into a shared resource pool
that multiple upstream subtasks can use and that scales elastically with
load, which maps directly onto what rpc operator provides: each instance
forming its own Pipelined Region, giving independent scaling and flexible
load balancing.
One question on the communication layer. The first version reuses Pekko
RPC, but the proposal only discusses latency overhead and QPS, not the size
of a single message. Pekko has a default cap on per-message size, and in AI
scenarios the context carried in a single call can be quite large
(multimodal inputs, long context, etc.). Has this case been considered? How
should large messages be handled, by chunking, by raising the limit, or by
going through a different transport?
Best,
Xintong
On Thu, May 28, 2026 at 7:35 PM Hongshun Wang <lo...@gmail.com>
wrote:
> Hi Yi,
>
> Thanks for the great work on FLIP-582. I've been reading through the
> proposal and have a few questions regarding the failover/recovery semantics
> and resource efficiency of RpcOperator Service. I'd appreciate any
> clarification.
>
>
> 1. Cost of recomputation due to stateless design
>
> The proposal states that "RPC services do not participate in
> checkpointing." This means that upon any data-plane failure and checkpoint
> rollback, all in-flight RPC requests are lost and must be re-triggered from
> scratch. For workloads where a single RPC invocation is expensive (e.g.,
> LLM inference taking seconds to minutes per request), this could lead to
> significant wasted computation.
>
> Has there been any consideration of a lightweight mechanism to avoid
> redundant recomputation — for example, a result cache on the service side
> keyed by request identity, or an optional "resumable request" protocol? Or
> is the position that this cost is acceptable given the target workload
> characteristics?
>
>
> 2. Cancelling in-flight RPC requests on data-plane failure
>
> When a data-plane operator fails and its Pipelined Region restarts, there
> may still be RPC requests in progress on the service side that were issued
> by the now-restarting data-plane tasks. Since those results will be
> discarded anyway (the data plane will re-issue them after recovery),
> continuing to process them wastes resources — particularly expensive GPU
> cycles.
>
> Is there a plan to provide an interface or mechanism for the data plane (or
> the framework) to proactively cancel outstanding RPC requests upon
> failure/restart? For example, a cancellation signal propagated from the
> RosClient to the RPC service, or a request-scoping mechanism tied to the
> checkpoint epoch, so that the service side can abort stale requests early.
>
>
> 3. Data-plane behavior when an RPC service subtask fails
>
> The proposal mentions that when an RPC service subtask fails, "requests are
> routed to other surviving instances." However, I'm curious about the
> behavior for requests that are already in-flight to the failed instance:
>
> - Does the data-plane operator have to wait for a timeout (as
> configured) before it knows the request has failed?
> - Is there any fast-failure notification mechanism (e.g.,
> connection-closed detection) that allows the RosClient to immediately
> fail
> over in-flight requests to other instances without waiting for the full
> timeout?
> - If the service has only one instance (parallelism = 1) and it fails,
> does the data plane simply block (with backpressure) until the instance
> recovers, or does it fail fast with UNAVAILABLE errors?
>
> A fast-failure path would be important for reducing end-to-end latency
> impact during service-side failures.
>
>
> 4. Resource utilization concerns with separate SlotSharingGroup
>
> The proposal states that "each RPC service automatically creates a separate
> SlotSharingGroup" and that this is "fully compatible with Flink's existing
> K8s/YARN resource management." However, in Flink's current resource model,
> all TaskManagers share the same configuration template (determined by
> global taskmanager.memory.* and CPU settings). Fine-Grained Resource
> Management controls how slots are sized within a TM, but does not enable
> heterogeneous TM specifications across the cluster.
>
> This raises a practical concern: if an RpcOperator requires GPU resources,
> the TM configuration must include GPU allocation. Since all TMs share the
> same template, this means every TM in the cluster would be provisioned with
> GPUs — even those exclusively hosting data-plane tasks that have no GPU
> requirement. This leads to significant GPU resource waste.
>
> Furthermore, the separate SlotSharingGroup design prevents co-locating
> RpcOperator and data-plane tasks on the same TaskManager. In practice,
> co-location (e.g., RpcOperator using the GPU while data-plane tasks use the
> CPU on the same GPU node) would improve overall resource utilization. The
> current design prioritizes isolation at the cost of efficiency.
>
> Has there been consideration of:
>
> - Supporting heterogeneous TaskManager specifications (different TM
> templates for different SlotSharingGroups) to avoid provisioning GPU on
> CPU-only TMs?(As a reference, KubeRay's Worker Group model addresses
> exactly this problem for Ray clusters. In KubeRay, a RayCluster defines
> multiple workerGroupSpecs, each with its own pod template, resource
> configuration, node selectors, and independent autoscaling bounds)
> - Allowing optional co-location of RpcOperator and data-plane slots on
> the same TM (in separate slots) for scenarios where utilization is
> prioritized over strict process-level isolation?
> - Or is there a dependency on other sub-FLIPs under FLIP-577 to address
> this gap in GPU resource scheduling?
>
> Thanks in advance for the clarification. Looking forward to the continued
> discussion.
>
>
> Best regards
>
> Hongshun
>
> On Thu, May 28, 2026 at 4:40 PM Guowei Ma <gu...@gmail.com> wrote:
>
> > HI,YI
> >
> > One more question I'd like to raise is around how the service behaves
> when
> > it's referenced by more than one job. If two jobs are submitted within
> the
> > same application and both reference the same RpcOperator service, is that
> > service shared across them, and what is the expected behavior in that
> case?
> > I'm also curious how this plays out for bounded (batch) jobs in
> particular:
> > if one job completes while the other is still running, what does the
> > service lifecycle look like — is the service torn down as soon as the
> first
> > job finishes, and if so, how does that affect the job that's still
> running?
> >
> > Best,
> > Guowei
> >
> >
> > On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
> >
> > > Hi,YI
> > >
> > > Thanks for putting together this FLIP — it's a well-thought-out
> proposal,
> > > and the RpcOperator Service is a really valuable addition for
> > AI/inference
> > > workloads on Flink. I have a few questions and suggestions:
> > >
> > >
> > > 1. Autoscaling
> > > How is scale-up/scale-down expressed in this design? Could you consider
> > > allowing concurrency to be configured as a (min, max) range? Currently
> it
> > > appears only min = max (i.e. fixed parallelism) is supported.
> > >
> > >
> > > 2. Async invocation & end-to-end examples
> > > The example shows a synchronous invocation style. I'd suggest also
> > > providing an asynchronous invocation example — or even just a scaffold
> > that
> > > exposes only the RpcOperator's handle and method. In addition, it would
> > be
> > > best to include a complete end-to-end example of invoking a model (CPU
> > > inference would be fine).
> > >
> > >
> > > 3. GPU as a resource
> > > GPU is surely a common/first-class resource. Accessing it here through
> an
> > > extend mechanism — wouldn't that mislead users? It would seem more
> > > natural to declare GPU as a standard resource rather than going through
> > an
> > > extension.
> > >
> > >
> > > 4. Serial method invocation
> > > Each method of an RpcOperator is invoked serially. I'd suggest stating
> > > this explicitly in both the documentation and the code comments, so
> that
> > > users — or an AI — can be clearly aware of it.
> > >
> > >
> > > 5. Why synchronous methods?
> > > Why is every method in RpcOperator a synchronous call? Does this imply
> > > that no model inference supports concurrent invocation?
> > >
> > >
> > > 6. SQL UDF usage
> > > How would SQL UDF users invoke this capability? I'd suggest providing a
> > > complete example for this as well.
> > >
> > >
> > > Thanks again for the great work!
> > >
> > > Best,
> > > Guowei
> > >
> > >
> > > On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
> > > wrote:
> > >
> > >> Hi Yi, and Flink Community,
> > >>
> > >> Thanks for bringing up this excellent proposal. I am fully in favor of
> > >> FLIP-582.
> > >>
> > >> In our production workloads, especially regarding large-scale AI
> > >> inference,
> > >> the tight coupling of the data plane and compute units has always
> been a
> > >> major pain point. If an inference subtask fails due to a GPU OOM or
> > driver
> > >> issue, triggering a global rollback is incredibly expensive since
> model
> > >> reloading takes minutes.
> > >>
> > >> The introduction of the RpcOperator Service as a first-class primitive
> > >> masterfully decouples the heavy inference tasks from the mainstream
> > >> topology. The fault isolation, independent scaling, and stateless
> design
> > >> perfectly match the requirements of modern AI-oriented data
> processing.
> > >>
> > >> This is a clean and robust architecture. Looking forward to seeing
> this
> > >> merged!
> > >>
> > >>
> > >> Best wishes,
> > >> Charles Zhang
> > >> from Apache InLong
> > >>
> > >>
> > >> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
> > >>
> > >> > Hi everyone,
> > >> >
> > >> >
> > >> >
> > >> > I would like to start a discussion on FLIP-582: Support RpcOperator
> > >> > Service [1].
> > >> >
> > >> >
> > >> > AI-oriented workloads like multimodal data processing and model
> > >> inference
> > >> > are
> > >> > growing rapidly in recent years. These workloads are characterized
> by
> > >> > expensive
> > >> > resources (GPUs) and high initialization costs (seconds to minutes
> for
> > >> > model
> > >> > loading). In today's Flink, embedding them in the data plane couples
> > >> their
> > >> > parallelism and failover with surrounding operators; deploying them
> as
> > >> > external
> > >> > services disconnects their lifecycle from the job and doubles
> > >> operational
> > >> > overhead.
> > >> >
> > >> >
> > >> > This FLIP introduces RpcOperator Service — a framework-level
> primitive
> > >> > that runs
> > >> > user-defined compute as RPC services in an independent Pipelined
> > Region
> > >> > within
> > >> > the Flink job. Because the service is isolated at the scheduling
> > level,
> > >> it
> > >> > can achieve
> > >> > fault isolation, independent scaling, and dedicated resource
> > allocation.
> > >> > As a native
> > >> > Flink primitive, it also lays the foundation for automatic flow
> > control,
> > >> > flexible load
> > >> > balancing, and coordinated auto-scaling — all without introducing
> > >> external
> > >> > infrastructure or additional operational burden.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Looking forward to your feedback and suggestions!
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > [1]
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Best Regards,
> > >> > Yi Zhang
> > >>
> > >
> >
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Hongshun,
Thank you for the in-depth review and the highly insightful questions! Your feedback has prompted us
to think more carefully about several important runtime aspects.
1. Cost of recomputation due to stateless design
This is a real and practical concern. For the of RPC services where request latency is manageable,
the cost of recomputation upon data-plane failover is acceptable. And the volume of replayed requests
can be effectively controlled by reducing the checkpoint interval, which limits the window of data that
needs to be reprocessed.
However, for long-running requests, the wasted computation deserves serious consideration. Server-side
result caching keyed by request identity is a promising direction. That said, caching alone may not be
sufficient: cache hits depend on replayed requests being routed to the **same service instance** that
processed the original request. This introduces a tension with load balancing — a sticky routing strategy
would improve cache hit rates but may compromise load distribution, making it suitable only for specific
scenarios where request affinity is acceptable.
I think this is best addressed as a relatively independent future optimization, combining server-side
caching with a configurable request routing strategy. For the current FLIP, the stateless design remains
the right starting point for its simplicity and broad applicability.
2. Cancelling in-flight RPC requests on data-plane failure
This is an excellent suggestion. Upon data-plane failure, the framework can indeed cancel requests that
are still **queued** on the service side. The mechanism works as follows: when a data-plane operator is
about to restart, it sends a cancellation signal to the service endpoints; upon receiving this signal, the
service removes the requests associated with that client to avoid unnecessary computation.
It is worth noting that requests **already being processed** cannot be effectively aborted. Cancelling the
potentially large backlog of queued requests is where the real value lies. I will include this cancellation
mechanism in the FLIP as part of the proposed design.
3. Data-plane behavior when an RPC service subtask fails
There is a fast-failure mechanism in place. The underlying transport layer typically detects connection
failures quickly — for example, the Pekko-based implementation is TCP-based, so when a service instance
fails and the connection drops, pending RPC calls on the client side receive an exception without waiting
for a timeout. The client can catch this exception and retry the request against other surviving instances.
When the service has only one instance (`parallelism = 1`) and that instance fails, the `ServiceRegistry` will
have no available endpoints. In this case, `RosClient` will **fail fast** with an `UNAVAILABLE` error rather
than blocking. The data-plane operator can then decide how to handle this based on its own error handling
logic.
4. Resource utilization concerns with separate SlotSharingGroup
This is indeed a fundamental issue at the Flink resource model level. Currently, Flink does not natively
support heterogeneous TaskManager specifications within a single cluster. That said, it is worth clarifying
that slots with different resource profiles **can** coexist on the same TaskManager through Flink's
[fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/finegrained_resource/).
However, the practical challenge is **resource fragmentation** — heterogeneous slots on a uniform TM
template can lead to underutilized resources.
I believe that supporting more flexible heterogeneous TM specifications is an important direction worth
pursuing. We plan to address this as a key topic in a dedicated FLIP focused on GPU resource scheduling,
where we can give it the thorough treatment it deserves — including heterogeneous TM templates,
GPU-aware placement, and node pool strategies.
Thank you again for these thoughtful and insightful questions!
Best,
Yi
At 2026-05-28 19:34:23, "Hongshun Wang" <lo...@gmail.com> wrote:
>Hi Yi,
>
>Thanks for the great work on FLIP-582. I've been reading through the
>proposal and have a few questions regarding the failover/recovery semantics
>and resource efficiency of RpcOperator Service. I'd appreciate any
>clarification.
>
>
>1. Cost of recomputation due to stateless design
>
>The proposal states that "RPC services do not participate in
>checkpointing." This means that upon any data-plane failure and checkpoint
>rollback, all in-flight RPC requests are lost and must be re-triggered from
>scratch. For workloads where a single RPC invocation is expensive (e.g.,
>LLM inference taking seconds to minutes per request), this could lead to
>significant wasted computation.
>
>Has there been any consideration of a lightweight mechanism to avoid
>redundant recomputation — for example, a result cache on the service side
>keyed by request identity, or an optional "resumable request" protocol? Or
>is the position that this cost is acceptable given the target workload
>characteristics?
>
>
>2. Cancelling in-flight RPC requests on data-plane failure
>
>When a data-plane operator fails and its Pipelined Region restarts, there
>may still be RPC requests in progress on the service side that were issued
>by the now-restarting data-plane tasks. Since those results will be
>discarded anyway (the data plane will re-issue them after recovery),
>continuing to process them wastes resources — particularly expensive GPU
>cycles.
>
>Is there a plan to provide an interface or mechanism for the data plane (or
>the framework) to proactively cancel outstanding RPC requests upon
>failure/restart? For example, a cancellation signal propagated from the
>RosClient to the RPC service, or a request-scoping mechanism tied to the
>checkpoint epoch, so that the service side can abort stale requests early.
>
>
>3. Data-plane behavior when an RPC service subtask fails
>
>The proposal mentions that when an RPC service subtask fails, "requests are
>routed to other surviving instances." However, I'm curious about the
>behavior for requests that are already in-flight to the failed instance:
>
> - Does the data-plane operator have to wait for a timeout (as
> configured) before it knows the request has failed?
> - Is there any fast-failure notification mechanism (e.g.,
> connection-closed detection) that allows the RosClient to immediately fail
> over in-flight requests to other instances without waiting for the full
> timeout?
> - If the service has only one instance (parallelism = 1) and it fails,
> does the data plane simply block (with backpressure) until the instance
> recovers, or does it fail fast with UNAVAILABLE errors?
>
>A fast-failure path would be important for reducing end-to-end latency
>impact during service-side failures.
>
>
>4. Resource utilization concerns with separate SlotSharingGroup
>
>The proposal states that "each RPC service automatically creates a separate
>SlotSharingGroup" and that this is "fully compatible with Flink's existing
>K8s/YARN resource management." However, in Flink's current resource model,
>all TaskManagers share the same configuration template (determined by
>global taskmanager.memory.* and CPU settings). Fine-Grained Resource
>Management controls how slots are sized within a TM, but does not enable
>heterogeneous TM specifications across the cluster.
>
>This raises a practical concern: if an RpcOperator requires GPU resources,
>the TM configuration must include GPU allocation. Since all TMs share the
>same template, this means every TM in the cluster would be provisioned with
>GPUs — even those exclusively hosting data-plane tasks that have no GPU
>requirement. This leads to significant GPU resource waste.
>
>Furthermore, the separate SlotSharingGroup design prevents co-locating
>RpcOperator and data-plane tasks on the same TaskManager. In practice,
>co-location (e.g., RpcOperator using the GPU while data-plane tasks use the
>CPU on the same GPU node) would improve overall resource utilization. The
>current design prioritizes isolation at the cost of efficiency.
>
>Has there been consideration of:
>
> - Supporting heterogeneous TaskManager specifications (different TM
> templates for different SlotSharingGroups) to avoid provisioning GPU on
> CPU-only TMs?(As a reference, KubeRay's Worker Group model addresses
> exactly this problem for Ray clusters. In KubeRay, a RayCluster defines
> multiple workerGroupSpecs, each with its own pod template, resource
> configuration, node selectors, and independent autoscaling bounds)
> - Allowing optional co-location of RpcOperator and data-plane slots on
> the same TM (in separate slots) for scenarios where utilization is
> prioritized over strict process-level isolation?
> - Or is there a dependency on other sub-FLIPs under FLIP-577 to address
> this gap in GPU resource scheduling?
>
>Thanks in advance for the clarification. Looking forward to the continued
>discussion.
>
>
>Best regards
>
>Hongshun
>
>On Thu, May 28, 2026 at 4:40 PM Guowei Ma <gu...@gmail.com> wrote:
>
>> HI,YI
>>
>> One more question I'd like to raise is around how the service behaves when
>> it's referenced by more than one job. If two jobs are submitted within the
>> same application and both reference the same RpcOperator service, is that
>> service shared across them, and what is the expected behavior in that case?
>> I'm also curious how this plays out for bounded (batch) jobs in particular:
>> if one job completes while the other is still running, what does the
>> service lifecycle look like — is the service torn down as soon as the first
>> job finishes, and if so, how does that affect the job that's still running?
>>
>> Best,
>> Guowei
>>
>>
>> On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
>>
>> > Hi,YI
>> >
>> > Thanks for putting together this FLIP — it's a well-thought-out proposal,
>> > and the RpcOperator Service is a really valuable addition for
>> AI/inference
>> > workloads on Flink. I have a few questions and suggestions:
>> >
>> >
>> > 1. Autoscaling
>> > How is scale-up/scale-down expressed in this design? Could you consider
>> > allowing concurrency to be configured as a (min, max) range? Currently it
>> > appears only min = max (i.e. fixed parallelism) is supported.
>> >
>> >
>> > 2. Async invocation & end-to-end examples
>> > The example shows a synchronous invocation style. I'd suggest also
>> > providing an asynchronous invocation example — or even just a scaffold
>> that
>> > exposes only the RpcOperator's handle and method. In addition, it would
>> be
>> > best to include a complete end-to-end example of invoking a model (CPU
>> > inference would be fine).
>> >
>> >
>> > 3. GPU as a resource
>> > GPU is surely a common/first-class resource. Accessing it here through an
>> > extend mechanism — wouldn't that mislead users? It would seem more
>> > natural to declare GPU as a standard resource rather than going through
>> an
>> > extension.
>> >
>> >
>> > 4. Serial method invocation
>> > Each method of an RpcOperator is invoked serially. I'd suggest stating
>> > this explicitly in both the documentation and the code comments, so that
>> > users — or an AI — can be clearly aware of it.
>> >
>> >
>> > 5. Why synchronous methods?
>> > Why is every method in RpcOperator a synchronous call? Does this imply
>> > that no model inference supports concurrent invocation?
>> >
>> >
>> > 6. SQL UDF usage
>> > How would SQL UDF users invoke this capability? I'd suggest providing a
>> > complete example for this as well.
>> >
>> >
>> > Thanks again for the great work!
>> >
>> > Best,
>> > Guowei
>> >
>> >
>> > On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
>> > wrote:
>> >
>> >> Hi Yi, and Flink Community,
>> >>
>> >> Thanks for bringing up this excellent proposal. I am fully in favor of
>> >> FLIP-582.
>> >>
>> >> In our production workloads, especially regarding large-scale AI
>> >> inference,
>> >> the tight coupling of the data plane and compute units has always been a
>> >> major pain point. If an inference subtask fails due to a GPU OOM or
>> driver
>> >> issue, triggering a global rollback is incredibly expensive since model
>> >> reloading takes minutes.
>> >>
>> >> The introduction of the RpcOperator Service as a first-class primitive
>> >> masterfully decouples the heavy inference tasks from the mainstream
>> >> topology. The fault isolation, independent scaling, and stateless design
>> >> perfectly match the requirements of modern AI-oriented data processing.
>> >>
>> >> This is a clean and robust architecture. Looking forward to seeing this
>> >> merged!
>> >>
>> >>
>> >> Best wishes,
>> >> Charles Zhang
>> >> from Apache InLong
>> >>
>> >>
>> >> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>> >>
>> >> > Hi everyone,
>> >> >
>> >> >
>> >> >
>> >> > I would like to start a discussion on FLIP-582: Support RpcOperator
>> >> > Service [1].
>> >> >
>> >> >
>> >> > AI-oriented workloads like multimodal data processing and model
>> >> inference
>> >> > are
>> >> > growing rapidly in recent years. These workloads are characterized by
>> >> > expensive
>> >> > resources (GPUs) and high initialization costs (seconds to minutes for
>> >> > model
>> >> > loading). In today's Flink, embedding them in the data plane couples
>> >> their
>> >> > parallelism and failover with surrounding operators; deploying them as
>> >> > external
>> >> > services disconnects their lifecycle from the job and doubles
>> >> operational
>> >> > overhead.
>> >> >
>> >> >
>> >> > This FLIP introduces RpcOperator Service — a framework-level primitive
>> >> > that runs
>> >> > user-defined compute as RPC services in an independent Pipelined
>> Region
>> >> > within
>> >> > the Flink job. Because the service is isolated at the scheduling
>> level,
>> >> it
>> >> > can achieve
>> >> > fault isolation, independent scaling, and dedicated resource
>> allocation.
>> >> > As a native
>> >> > Flink primitive, it also lays the foundation for automatic flow
>> control,
>> >> > flexible load
>> >> > balancing, and coordinated auto-scaling — all without introducing
>> >> external
>> >> > infrastructure or additional operational burden.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Looking forward to your feedback and suggestions!
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > [1]
>> >> >
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Best Regards,
>> >> > Yi Zhang
>> >>
>> >
>>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Hongshun Wang <lo...@gmail.com>.
Hi Yi,
Thanks for the great work on FLIP-582. I've been reading through the
proposal and have a few questions regarding the failover/recovery semantics
and resource efficiency of RpcOperator Service. I'd appreciate any
clarification.
1. Cost of recomputation due to stateless design
The proposal states that "RPC services do not participate in
checkpointing." This means that upon any data-plane failure and checkpoint
rollback, all in-flight RPC requests are lost and must be re-triggered from
scratch. For workloads where a single RPC invocation is expensive (e.g.,
LLM inference taking seconds to minutes per request), this could lead to
significant wasted computation.
Has there been any consideration of a lightweight mechanism to avoid
redundant recomputation — for example, a result cache on the service side
keyed by request identity, or an optional "resumable request" protocol? Or
is the position that this cost is acceptable given the target workload
characteristics?
2. Cancelling in-flight RPC requests on data-plane failure
When a data-plane operator fails and its Pipelined Region restarts, there
may still be RPC requests in progress on the service side that were issued
by the now-restarting data-plane tasks. Since those results will be
discarded anyway (the data plane will re-issue them after recovery),
continuing to process them wastes resources — particularly expensive GPU
cycles.
Is there a plan to provide an interface or mechanism for the data plane (or
the framework) to proactively cancel outstanding RPC requests upon
failure/restart? For example, a cancellation signal propagated from the
RosClient to the RPC service, or a request-scoping mechanism tied to the
checkpoint epoch, so that the service side can abort stale requests early.
3. Data-plane behavior when an RPC service subtask fails
The proposal mentions that when an RPC service subtask fails, "requests are
routed to other surviving instances." However, I'm curious about the
behavior for requests that are already in-flight to the failed instance:
- Does the data-plane operator have to wait for a timeout (as
configured) before it knows the request has failed?
- Is there any fast-failure notification mechanism (e.g.,
connection-closed detection) that allows the RosClient to immediately fail
over in-flight requests to other instances without waiting for the full
timeout?
- If the service has only one instance (parallelism = 1) and it fails,
does the data plane simply block (with backpressure) until the instance
recovers, or does it fail fast with UNAVAILABLE errors?
A fast-failure path would be important for reducing end-to-end latency
impact during service-side failures.
4. Resource utilization concerns with separate SlotSharingGroup
The proposal states that "each RPC service automatically creates a separate
SlotSharingGroup" and that this is "fully compatible with Flink's existing
K8s/YARN resource management." However, in Flink's current resource model,
all TaskManagers share the same configuration template (determined by
global taskmanager.memory.* and CPU settings). Fine-Grained Resource
Management controls how slots are sized within a TM, but does not enable
heterogeneous TM specifications across the cluster.
This raises a practical concern: if an RpcOperator requires GPU resources,
the TM configuration must include GPU allocation. Since all TMs share the
same template, this means every TM in the cluster would be provisioned with
GPUs — even those exclusively hosting data-plane tasks that have no GPU
requirement. This leads to significant GPU resource waste.
Furthermore, the separate SlotSharingGroup design prevents co-locating
RpcOperator and data-plane tasks on the same TaskManager. In practice,
co-location (e.g., RpcOperator using the GPU while data-plane tasks use the
CPU on the same GPU node) would improve overall resource utilization. The
current design prioritizes isolation at the cost of efficiency.
Has there been consideration of:
- Supporting heterogeneous TaskManager specifications (different TM
templates for different SlotSharingGroups) to avoid provisioning GPU on
CPU-only TMs?(As a reference, KubeRay's Worker Group model addresses
exactly this problem for Ray clusters. In KubeRay, a RayCluster defines
multiple workerGroupSpecs, each with its own pod template, resource
configuration, node selectors, and independent autoscaling bounds)
- Allowing optional co-location of RpcOperator and data-plane slots on
the same TM (in separate slots) for scenarios where utilization is
prioritized over strict process-level isolation?
- Or is there a dependency on other sub-FLIPs under FLIP-577 to address
this gap in GPU resource scheduling?
Thanks in advance for the clarification. Looking forward to the continued
discussion.
Best regards
Hongshun
On Thu, May 28, 2026 at 4:40 PM Guowei Ma <gu...@gmail.com> wrote:
> HI,YI
>
> One more question I'd like to raise is around how the service behaves when
> it's referenced by more than one job. If two jobs are submitted within the
> same application and both reference the same RpcOperator service, is that
> service shared across them, and what is the expected behavior in that case?
> I'm also curious how this plays out for bounded (batch) jobs in particular:
> if one job completes while the other is still running, what does the
> service lifecycle look like — is the service torn down as soon as the first
> job finishes, and if so, how does that affect the job that's still running?
>
> Best,
> Guowei
>
>
> On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
>
> > Hi,YI
> >
> > Thanks for putting together this FLIP — it's a well-thought-out proposal,
> > and the RpcOperator Service is a really valuable addition for
> AI/inference
> > workloads on Flink. I have a few questions and suggestions:
> >
> >
> > 1. Autoscaling
> > How is scale-up/scale-down expressed in this design? Could you consider
> > allowing concurrency to be configured as a (min, max) range? Currently it
> > appears only min = max (i.e. fixed parallelism) is supported.
> >
> >
> > 2. Async invocation & end-to-end examples
> > The example shows a synchronous invocation style. I'd suggest also
> > providing an asynchronous invocation example — or even just a scaffold
> that
> > exposes only the RpcOperator's handle and method. In addition, it would
> be
> > best to include a complete end-to-end example of invoking a model (CPU
> > inference would be fine).
> >
> >
> > 3. GPU as a resource
> > GPU is surely a common/first-class resource. Accessing it here through an
> > extend mechanism — wouldn't that mislead users? It would seem more
> > natural to declare GPU as a standard resource rather than going through
> an
> > extension.
> >
> >
> > 4. Serial method invocation
> > Each method of an RpcOperator is invoked serially. I'd suggest stating
> > this explicitly in both the documentation and the code comments, so that
> > users — or an AI — can be clearly aware of it.
> >
> >
> > 5. Why synchronous methods?
> > Why is every method in RpcOperator a synchronous call? Does this imply
> > that no model inference supports concurrent invocation?
> >
> >
> > 6. SQL UDF usage
> > How would SQL UDF users invoke this capability? I'd suggest providing a
> > complete example for this as well.
> >
> >
> > Thanks again for the great work!
> >
> > Best,
> > Guowei
> >
> >
> > On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
> > wrote:
> >
> >> Hi Yi, and Flink Community,
> >>
> >> Thanks for bringing up this excellent proposal. I am fully in favor of
> >> FLIP-582.
> >>
> >> In our production workloads, especially regarding large-scale AI
> >> inference,
> >> the tight coupling of the data plane and compute units has always been a
> >> major pain point. If an inference subtask fails due to a GPU OOM or
> driver
> >> issue, triggering a global rollback is incredibly expensive since model
> >> reloading takes minutes.
> >>
> >> The introduction of the RpcOperator Service as a first-class primitive
> >> masterfully decouples the heavy inference tasks from the mainstream
> >> topology. The fault isolation, independent scaling, and stateless design
> >> perfectly match the requirements of modern AI-oriented data processing.
> >>
> >> This is a clean and robust architecture. Looking forward to seeing this
> >> merged!
> >>
> >>
> >> Best wishes,
> >> Charles Zhang
> >> from Apache InLong
> >>
> >>
> >> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
> >>
> >> > Hi everyone,
> >> >
> >> >
> >> >
> >> > I would like to start a discussion on FLIP-582: Support RpcOperator
> >> > Service [1].
> >> >
> >> >
> >> > AI-oriented workloads like multimodal data processing and model
> >> inference
> >> > are
> >> > growing rapidly in recent years. These workloads are characterized by
> >> > expensive
> >> > resources (GPUs) and high initialization costs (seconds to minutes for
> >> > model
> >> > loading). In today's Flink, embedding them in the data plane couples
> >> their
> >> > parallelism and failover with surrounding operators; deploying them as
> >> > external
> >> > services disconnects their lifecycle from the job and doubles
> >> operational
> >> > overhead.
> >> >
> >> >
> >> > This FLIP introduces RpcOperator Service — a framework-level primitive
> >> > that runs
> >> > user-defined compute as RPC services in an independent Pipelined
> Region
> >> > within
> >> > the Flink job. Because the service is isolated at the scheduling
> level,
> >> it
> >> > can achieve
> >> > fault isolation, independent scaling, and dedicated resource
> allocation.
> >> > As a native
> >> > Flink primitive, it also lays the foundation for automatic flow
> control,
> >> > flexible load
> >> > balancing, and coordinated auto-scaling — all without introducing
> >> external
> >> > infrastructure or additional operational burden.
> >> >
> >> >
> >> >
> >> >
> >> > Looking forward to your feedback and suggestions!
> >> >
> >> >
> >> >
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Best Regards,
> >> > Yi Zhang
> >>
> >
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Gyula,
Thanks for the feedback. I've added a new "Limitations and Usage Guide" section
(placed immediately after the Motivation) to address your concerns:
1. Limitations called out first: No state management, no temporal operations, no
exactly-once semantics.
2. Processing guarantees clarified: Data plane + ROS job provides at-least-once (via
checkpoint replay); ROS-only job provides no fault-tolerance guarantees.
3. When to Use / When Not to Use: Workloads must be compatible with the limitations
above and ROS is recommended only when the benefits of isolation outweigh the RPC
overhead. It is scoped to decoupling expensive, stateless compute from the data plane
— not a general-purpose service framework or a replacement for any existing API.
Please see the updated document for details. Thanks again for driving greater clarity
on the scope.
Best,
Yi
At 2026-06-12 17:25:59, "Gyula Fóra" <gy...@gmail.com> wrote:
>Hi Yi!
>I understand the overall motivation but I think we need concise and direct
>guidance, not only when we recommend using the RpcOperator but when we dont.
>
>Features like "fault isolation and flexible scaling" sound too powerful and
>users will tend to gravitate to these features if they promise a silver
>bullet against Flink's current limitations.
>Especially with the possibility of RPC-only jobs the whole datastream api
>may be questionable for users for stateless jobs.
>
>We have to clearly enumerate the cases when we recommend the rpc operator
>but also when we dont (this is very important)
>We have to explicitly and in the primary position call out what it doesn't
>provide: such as state.
>Are there any difference in processing semantics / guarantees? Can it even
>provide at least once guarantees for an RPC only job?
>
>Based on these we should aim to completely clarify the scope (not the
>features) to the users.
>
>Cheers
>Gyula
>
>
>On Fri, Jun 12, 2026 at 4:30 AM Yi Zhang <zh...@163.com> wrote:
>
>> Hi Guowei,
>>
>>
>> Thank you for your patience — I'm sorry for the delayed reply. I really
>> appreciate your
>> follow-up questions; they point to two key design choices that deserve
>> thorough
>> consideration.
>>
>>
>> 1. Auto-scaling & parallelism API
>> I believe we are in agreement that auto-scaling logic will not be included
>> in this FLIP.
>> Given that, I wonder whether introducing `min` and `max` parameters at
>> this stage
>> might actually confuse users — they would see an API that accepts a range
>> but offers
>> no runtime behavior to act on it.
>> I'd like to clarify how I think about the `parallelism` parameter: it does
>> not necessarily
>> represent a "fixed" parallelism, but rather an **initial parallelism** —
>> the service starts
>> with this level of concurrency and may be adjusted during subsequent
>> execution. Of
>> course, such adjustment would indeed require bounds. Would it be
>> acceptable to
>> introduce them when we actually bring in the auto-scaling capability? This
>> way, the
>> initial API remains simple and intuitive.
>>
>>
>> 2. Lifecycle semantics and API expression
>> This is a great question, and it actually connects closely to the **Shared
>> RPC Service**
>> task I mentioned in the Follow-up section. After further discussion and
>> reflection, I now
>> believe it would be better to include the shared service capability in
>> this FLIP as well.
>> The idea is to introduce a separate API:
>> `env.startSharedRpcOperatorService(...)`, which
>> is clearly distinct from `env.addRpcOperatorService(...)`. The semantics
>> would be:
>> a. `addRpcOperatorService` retains its **per-job** semantics, similar to
>> how
>> transformations are cleared after `execute()`, services added via this API
>> are also scoped
>> to a single job execution and cleaned up afterward.
>> b. `startSharedRpcOperatorService` starts a service that lives
>> independently of any
>> particular job, accessible by multiple jobs within the same application.
>>
>>
>> Regarding your suggestion of `ds.query(new RPCOperatorService)` — it's a
>> nice idea
>> for expressing per-job ownership, but I see one limitation: since the
>> service is bound to
>> a specific DataStream operation, every `query` call would instantiate a
>> new, independent
>> service. This makes it difficult to share a single service across multiple
>> operators within
>> the same job, which is a key use case we want to support.
>>
>>
>> With the `addRpcOperatorService` / `startSharedRpcOperatorService` pair,
>> both per-job
>> and shared scenarios are covered through a unified and explicit API. Do
>> you think this
>> approach works?
>>
>>
>> If you are aligned with this approach, I will update the FLIP accordingly.
>> Thank you again
>> for raising these important points!
>>
>>
>> Best,
>> Yi
>>
>>
>>
>> At 2026-06-02 14:27:26, "Guowei Ma" <gu...@gmail.com> wrote:
>> >Hi, YI
>> >Thanks a lot for the explanation and the update. I still have two small
>> >questions I'd like to get your thoughts on:
>> >
>> >*1. On auto-scaling*
>> >
>> >I think auto-scaling should be a first-class citizen in Flink as a whole —
>> >and RpcOperator was proposed precisely to make this achievable for GPU
>> >workloads. For that reason, baking a fixed parallelism into the API as the
>> >first step doesn't feel right to me; put simply, I don't think we should
>> >introduce an API that locks parallelism to a single value.
>> >
>> >That said, I fully agree that a complete auto-scaling implementation is
>> >complex and isn't the focus of this FLIP. So my suggestion is that the
>> >first version could just support the degenerate case — e.g. min = max —
>> >without committing the API itself to fixed parallelism.
>> >
>> >*2. On the lifecycle question*
>> >
>> >a. This is actually tied to how we express the API. What I've had in mind
>> >is something like ds.query(new RPCOperatorService), since it leaves no
>> room
>> >for ambiguity: the service clearly belongs to a single job, and every job
>> >has its own.
>> >
>> >b. Consider this scenario: a Flink application-mode deployment containing
>> >two jobs, where each job needs its own independent RPC service. With your
>> >current design, would I have to call addRpcService twice?
>> >
>> >c. If it's StreamExecutionEnvironment that interacts with the RPC
>> operator,
>> >that feels more like something shared *across* jobs — analogous to
>> >registering a UDF in the Table API, where multiple jobs can use it. That
>> >seems to sit in tension with the per-job semantics above.
>> >Best,
>> >Guowei
>> >
>> >
>> >On Tue, Jun 2, 2026 at 11:16 AM Gen Luo <lu...@gmail.com> wrote:
>> >
>> >> Hi Yi,
>> >>
>> >> Thanks for this impressive proposal. Adding an RPC service deployment
>> >> primitive can greatly expand Flink's capability scope and flexibility,
>> with
>> >> clear benefits for AI workloads and agent applications. It can also
>> play a
>> >> role in scenarios like state management or data lake compaction —
>> relevant
>> >> to but decoupled from the data pipeline. This is a very promising
>> >> direction. That said, I still have some questions.
>> >>
>> >> 1. My biggest concern is heterogeneous resources, echoing Hongshun's Q4.
>> >> Flink today uses homogeneous TM resources sliced for different task
>> >> requirements, but decoupling GPU/CPU resources and enabling flexible
>> >> independent scaling is one of the core values of this FLIP — and the
>> >> current resource allocation approach cannot support this. We likely need
>> >> new implementations for the resource allocation component.
>> >>
>> >> 2. Some components are described only at the idea level without
>> >> implementation details. Hopefully these can be supplemented later or
>> >> discussed in dedicated threads. A few worth discussing:
>> >>
>> >> - RpcTask: It's currently described as a simple single-threaded loop,
>> >> with no plan to use Mailbox or OperatorChain. But if we want to support
>> >> async, checkpoint, etc. later, introducing Mailbox would be valuable —
>> >> handle the input queue as defaultAction, with async/CP/system messages
>> as
>> >> mails. This keeps semantics consistent with StreamTask and reduces
>> >> maintenance burden. System events like FO-triggered request cancellation
>> >> can also be prioritized via mailbox. OperatorChain is what
>> >> SubtaskCheckpointCoordinator directly contacts, so having RpcOperator as
>> >> the only node in the chain wouldn't add much complexity and lowers the
>> cost
>> >> when CP support comes.
>> >>
>> >> - Communication implementation: Will it reuse the Pekko + proxy
>> >> mechanism? Pekko has risks with large message bodies (OOM, silently drop
>> >> oversized frame, etc.). Since ROS is user-facing, we can't fully avoid
>> >> this, and large-model scenarios often carry sizable context — needs
>> careful
>> >> handling. Also, the current design doesn't include table-function-style
>> >> interfaces for unfolding data, so a large unfolded result has to be
>> >> returned as a single response, inflating per-request payload size and
>> >> amplifying the same risk.
>> >>
>> >> - Client-side implementation: Only the interface is described, no
>> >> implementation details. Checkpoint is the prominent issue — since ROS is
>> >> stateless by design, state recording falls to the client. Do we go with
>> >> drain or record? Drain may take a long time; record on a keyed stream
>> may
>> >> cause request distribution confusion after parallelism changes.
>> >>
>> >> 3. ROS as container, not service: Is this allowed? If the system itself
>> has
>> >> service capabilities — especially non-Java stacks (agents, LLMs, also
>> >> Python remote functions) — forwarding requests through the Flink Java
>> layer
>> >> adds unnecessary overhead. In that case RpcOperator would only handle
>> >> lifecycle, while the framework manages resource scheduling and service
>> >> discovery without carrying business traffic.
>> >>
>> >> nit: addRpcService is inconsistent with other places that use
>> RpcOperator /
>> >> the ros keyword. Worth unifying.
>> >>
>> >> Best,
>> >> Gen
>> >>
>> >> On Mon, Jun 1, 2026 at 2:02 PM Yi Zhang <zh...@163.com> wrote:
>> >>
>> >> > Hi Guowei,
>> >> >
>> >> >
>> >> > Thanks for the thoughtful review and valuable feedback! Your questions
>> >> are
>> >> > insightful
>> >> > and have directly driven refinements to the proposal. Here are my
>> >> > responses to each
>> >> > point:
>> >> >
>> >> >
>> >> > 1. Autoscaling
>> >> > Supporting a (min, max) concurrency range is a great idea. However,
>> >> > auto-scaling
>> >> > involves much more than just the API surface — it requires discussion
>> >> > around
>> >> > scaling policies, coordination with the scheduler, metric-driven
>> >> triggers,
>> >> > and more.
>> >> > To keep this FLIP focused, I'd like to defer that to a follow-up.
>> Fixed
>> >> > parallelism is
>> >> > a common and necessary capability on its own, and the first version
>> will
>> >> > support
>> >> > that. The interfaces can be extended later to include auto-scaling
>> >> support
>> >> > without
>> >> > breaking changes.
>> >> >
>> >> >
>> >> > 2. Async invocation & end-to-end examples
>> >> > This is a good point. The `RosClient` already provides an async
>> >> interface,
>> >> > so using
>> >> > it with `AsyncDataStream` is indeed more natural and idiomatic. I will
>> >> add
>> >> > an
>> >> > asynchronous invocation example in the proposal. Regarding a complete
>> >> > end-to-end
>> >> > inference example: at the Java API level, it will still be primarily
>> >> > conceptual, since
>> >> > practical model inference examples really need Python API support
>> (e.g.,
>> >> > loading a
>> >> > PyTorch or ONNX model). I will refine the existing example to make the
>> >> > overall
>> >> > invocation chain as close to a real-world scenario as possible.
>> >> >
>> >> >
>> >> > 3. GPU as a resource
>> >> > This is indeed a key concern. The `extendedResource` API in the
>> original
>> >> > proposal
>> >> > comes from Flink's existing resource model (introduced by FLIP-108).
>> >> > However, I
>> >> > agree that GPU deserves first-class treatment rather than being
>> accessed
>> >> > through a
>> >> > generic extension mechanism. Promoting GPU to a first-class resource
>> will
>> >> > involve
>> >> > changes to both the API and the scheduling layer, which we plan to
>> >> address
>> >> > in a
>> >> > dedicated FLIP. For now, I will remove the `extendedResource` API from
>> >> > `RosDescriptor` to keep this FLIP focused on the RPC service primitive
>> >> > itself. The
>> >> > resource declaration API can be extended once the GPU-specific FLIP
>> is in
>> >> > place.
>> >> >
>> >> >
>> >> > 4. Serial method invocation
>> >> > Agreed. I will make the serial invocation semantics more prominent —
>> both
>> >> > in the
>> >> > FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
>> >> >
>> >> >
>> >> > 5. Why synchronous methods?
>> >> > This is a key question. The initial design chose synchronous methods
>> for
>> >> > simplicity
>> >> > — users don't need to worry about thread safety, and concurrency is
>> >> > achieved by
>> >> > scaling out instances (parallelism). However, I recognize that
>> >> > asynchronous
>> >> > processing is a critical requirement for certain scenarios (e.g., when
>> >> the
>> >> > RPC service
>> >> > performs expensive operations that can be parallelized internally). I
>> am
>> >> > planning to
>> >> > extend the current design to support asynchronous method execution:
>> if a
>> >> > user's
>> >> > method returns a `CompletableFuture`, the framework will recognize it
>> and
>> >> > wait for
>> >> > the future to complete before sending the response, without blocking
>> the
>> >> > invoke
>> >> > thread. This allows the invoke thread to continue dispatching
>> subsequent
>> >> > requests,
>> >> > while the user controls internal concurrency. Importantly,
>> >> > `processRequest` calls
>> >> > remain serial — the async behavior is purely opt-in.
>> >> >
>> >> >
>> >> > 6. SQL UDF usage
>> >> > The current design is primarily oriented toward the DataStream API,
>> as it
>> >> > provides
>> >> > the foundational building blocks. I agree that SQL and Table API
>> support
>> >> > is an
>> >> > important aspect to consider as well. On the invocation side, both
>> Table
>> >> > API and
>> >> > SQL UDFs share the same runtime path through `FunctionContext`, so
>> >> > enabling
>> >> > RPC service access from UDFs is straightforward — I will add a
>> >> > `getRosClient`
>> >> > method to `FunctionContext` and include that change in this FLIP.
>> >> > The real challenge for SQL lies on the definition side: how to define
>> and
>> >> > register
>> >> > an RPC service using pure SQL syntax, which would require SQL grammar
>> >> > extensions and planner changes. I think that part would be better
>> >> > addressed in
>> >> > a dedicated follow-up FLIP to give it the attention it deserves.
>> >> >
>> >> >
>> >> > 7. Service sharing across jobs & lifecycle
>> >> > In this FLIP, the RPC service is defined as part of the job (via
>> >> > `env.addRpcService()`
>> >> > before `env.execute()`). It cannot be accessed across jobs, and it is
>> >> > released when
>> >> > the job reaches a terminal state. I believe your question is more
>> >> relevant
>> >> > to the
>> >> > "Shared RPC Service" mentioned in the Follow-up Tasks section. While
>> it's
>> >> > out of
>> >> > scope for this FLIP, let me briefly describe the envisioned design: a
>> >> > shared RPC
>> >> > service would run as an independent job within the same Application,
>> >> > accessible by
>> >> > all jobs in that Application. It would terminate when the Application
>> >> > ends, or could be
>> >> > stopped manually via a user-facing API.
>> >> > The embedded (per-job) service model discussed in this FLIP is
>> simpler to
>> >> > manage
>> >> > and well-suited for many streaming scenarios, which is why we focus
>> on it
>> >> > first.
>> >> > Importantly, both the embedded and shared modes will reuse the same
>> set
>> >> of
>> >> > deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`,
>> etc.)
>> >> > — the
>> >> > difference is only in lifecycle management and visibility scope.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > I hope the above addresses your questions. I will update the proposal
>> to
>> >> > reflect the
>> >> > changes mentioned above. Please feel free to follow up if you have any
>> >> > further
>> >> > questions or suggestions.
>> >> >
>> >> >
>> >> > Best,
>> >> > Yi
>> >> >
>> >> >
>> >> >
>> >> > At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
>> >> > >HI,YI
>> >> > >
>> >> > >One more question I'd like to raise is around how the service behaves
>> >> when
>> >> > >it's referenced by more than one job. If two jobs are submitted
>> within
>> >> the
>> >> > >same application and both reference the same RpcOperator service, is
>> >> that
>> >> > >service shared across them, and what is the expected behavior in that
>> >> > case?
>> >> > >I'm also curious how this plays out for bounded (batch) jobs in
>> >> > particular:
>> >> > >if one job completes while the other is still running, what does the
>> >> > >service lifecycle look like — is the service torn down as soon as the
>> >> > first
>> >> > >job finishes, and if so, how does that affect the job that's still
>> >> > running?
>> >> > >
>> >> > >Best,
>> >> > >Guowei
>> >> > >
>> >> > >
>> >> > >On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com>
>> wrote:
>> >> > >
>> >> > >> Hi,YI
>> >> > >>
>> >> > >> Thanks for putting together this FLIP — it's a well-thought-out
>> >> > proposal,
>> >> > >> and the RpcOperator Service is a really valuable addition for
>> >> > AI/inference
>> >> > >> workloads on Flink. I have a few questions and suggestions:
>> >> > >>
>> >> > >>
>> >> > >> 1. Autoscaling
>> >> > >> How is scale-up/scale-down expressed in this design? Could you
>> >> consider
>> >> > >> allowing concurrency to be configured as a (min, max) range?
>> Currently
>> >> > it
>> >> > >> appears only min = max (i.e. fixed parallelism) is supported.
>> >> > >>
>> >> > >>
>> >> > >> 2. Async invocation & end-to-end examples
>> >> > >> The example shows a synchronous invocation style. I'd suggest also
>> >> > >> providing an asynchronous invocation example — or even just a
>> scaffold
>> >> > that
>> >> > >> exposes only the RpcOperator's handle and method. In addition, it
>> >> would
>> >> > be
>> >> > >> best to include a complete end-to-end example of invoking a model
>> (CPU
>> >> > >> inference would be fine).
>> >> > >>
>> >> > >>
>> >> > >> 3. GPU as a resource
>> >> > >> GPU is surely a common/first-class resource. Accessing it here
>> through
>> >> > an
>> >> > >> extend mechanism — wouldn't that mislead users? It would seem more
>> >> > >> natural to declare GPU as a standard resource rather than going
>> >> through
>> >> > an
>> >> > >> extension.
>> >> > >>
>> >> > >>
>> >> > >> 4. Serial method invocation
>> >> > >> Each method of an RpcOperator is invoked serially. I'd suggest
>> stating
>> >> > >> this explicitly in both the documentation and the code comments, so
>> >> that
>> >> > >> users — or an AI — can be clearly aware of it.
>> >> > >>
>> >> > >>
>> >> > >> 5. Why synchronous methods?
>> >> > >> Why is every method in RpcOperator a synchronous call? Does this
>> imply
>> >> > >> that no model inference supports concurrent invocation?
>> >> > >>
>> >> > >>
>> >> > >> 6. SQL UDF usage
>> >> > >> How would SQL UDF users invoke this capability? I'd suggest
>> providing
>> >> a
>> >> > >> complete example for this as well.
>> >> > >>
>> >> > >>
>> >> > >> Thanks again for the great work!
>> >> > >>
>> >> > >> Best,
>> >> > >> Guowei
>> >> > >>
>> >> > >>
>> >> > >> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <
>> dockerzhang@apache.org
>> >> >
>> >> > >> wrote:
>> >> > >>
>> >> > >>> Hi Yi, and Flink Community,
>> >> > >>>
>> >> > >>> Thanks for bringing up this excellent proposal. I am fully in
>> favor
>> >> of
>> >> > >>> FLIP-582.
>> >> > >>>
>> >> > >>> In our production workloads, especially regarding large-scale AI
>> >> > >>> inference,
>> >> > >>> the tight coupling of the data plane and compute units has always
>> >> been
>> >> > a
>> >> > >>> major pain point. If an inference subtask fails due to a GPU OOM
>> or
>> >> > driver
>> >> > >>> issue, triggering a global rollback is incredibly expensive since
>> >> model
>> >> > >>> reloading takes minutes.
>> >> > >>>
>> >> > >>> The introduction of the RpcOperator Service as a first-class
>> >> primitive
>> >> > >>> masterfully decouples the heavy inference tasks from the
>> mainstream
>> >> > >>> topology. The fault isolation, independent scaling, and stateless
>> >> > design
>> >> > >>> perfectly match the requirements of modern AI-oriented data
>> >> processing.
>> >> > >>>
>> >> > >>> This is a clean and robust architecture. Looking forward to seeing
>> >> this
>> >> > >>> merged!
>> >> > >>>
>> >> > >>>
>> >> > >>> Best wishes,
>> >> > >>> Charles Zhang
>> >> > >>> from Apache InLong
>> >> > >>>
>> >> > >>>
>> >> > >>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>> >> > >>>
>> >> > >>> > Hi everyone,
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > I would like to start a discussion on FLIP-582: Support
>> RpcOperator
>> >> > >>> > Service [1].
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > AI-oriented workloads like multimodal data processing and model
>> >> > >>> inference
>> >> > >>> > are
>> >> > >>> > growing rapidly in recent years. These workloads are
>> characterized
>> >> by
>> >> > >>> > expensive
>> >> > >>> > resources (GPUs) and high initialization costs (seconds to
>> minutes
>> >> > for
>> >> > >>> > model
>> >> > >>> > loading). In today's Flink, embedding them in the data plane
>> >> couples
>> >> > >>> their
>> >> > >>> > parallelism and failover with surrounding operators; deploying
>> them
>> >> > as
>> >> > >>> > external
>> >> > >>> > services disconnects their lifecycle from the job and doubles
>> >> > >>> operational
>> >> > >>> > overhead.
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > This FLIP introduces RpcOperator Service — a framework-level
>> >> > primitive
>> >> > >>> > that runs
>> >> > >>> > user-defined compute as RPC services in an independent Pipelined
>> >> > Region
>> >> > >>> > within
>> >> > >>> > the Flink job. Because the service is isolated at the scheduling
>> >> > level,
>> >> > >>> it
>> >> > >>> > can achieve
>> >> > >>> > fault isolation, independent scaling, and dedicated resource
>> >> > allocation.
>> >> > >>> > As a native
>> >> > >>> > Flink primitive, it also lays the foundation for automatic flow
>> >> > control,
>> >> > >>> > flexible load
>> >> > >>> > balancing, and coordinated auto-scaling — all without
>> introducing
>> >> > >>> external
>> >> > >>> > infrastructure or additional operational burden.
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > Looking forward to your feedback and suggestions!
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > [1]
>> >> > >>> >
>> >> > >>> >
>> >> > >>>
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > Best Regards,
>> >> > >>> > Yi Zhang
>> >> > >>>
>> >> > >>
>> >> >
>> >>
>>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Yi!
I understand the overall motivation but I think we need concise and direct
guidance, not only when we recommend using the RpcOperator but when we dont.
Features like "fault isolation and flexible scaling" sound too powerful and
users will tend to gravitate to these features if they promise a silver
bullet against Flink's current limitations.
Especially with the possibility of RPC-only jobs the whole datastream api
may be questionable for users for stateless jobs.
We have to clearly enumerate the cases when we recommend the rpc operator
but also when we dont (this is very important)
We have to explicitly and in the primary position call out what it doesn't
provide: such as state.
Are there any difference in processing semantics / guarantees? Can it even
provide at least once guarantees for an RPC only job?
Based on these we should aim to completely clarify the scope (not the
features) to the users.
Cheers
Gyula
On Fri, Jun 12, 2026 at 4:30 AM Yi Zhang <zh...@163.com> wrote:
> Hi Guowei,
>
>
> Thank you for your patience — I'm sorry for the delayed reply. I really
> appreciate your
> follow-up questions; they point to two key design choices that deserve
> thorough
> consideration.
>
>
> 1. Auto-scaling & parallelism API
> I believe we are in agreement that auto-scaling logic will not be included
> in this FLIP.
> Given that, I wonder whether introducing `min` and `max` parameters at
> this stage
> might actually confuse users — they would see an API that accepts a range
> but offers
> no runtime behavior to act on it.
> I'd like to clarify how I think about the `parallelism` parameter: it does
> not necessarily
> represent a "fixed" parallelism, but rather an **initial parallelism** —
> the service starts
> with this level of concurrency and may be adjusted during subsequent
> execution. Of
> course, such adjustment would indeed require bounds. Would it be
> acceptable to
> introduce them when we actually bring in the auto-scaling capability? This
> way, the
> initial API remains simple and intuitive.
>
>
> 2. Lifecycle semantics and API expression
> This is a great question, and it actually connects closely to the **Shared
> RPC Service**
> task I mentioned in the Follow-up section. After further discussion and
> reflection, I now
> believe it would be better to include the shared service capability in
> this FLIP as well.
> The idea is to introduce a separate API:
> `env.startSharedRpcOperatorService(...)`, which
> is clearly distinct from `env.addRpcOperatorService(...)`. The semantics
> would be:
> a. `addRpcOperatorService` retains its **per-job** semantics, similar to
> how
> transformations are cleared after `execute()`, services added via this API
> are also scoped
> to a single job execution and cleaned up afterward.
> b. `startSharedRpcOperatorService` starts a service that lives
> independently of any
> particular job, accessible by multiple jobs within the same application.
>
>
> Regarding your suggestion of `ds.query(new RPCOperatorService)` — it's a
> nice idea
> for expressing per-job ownership, but I see one limitation: since the
> service is bound to
> a specific DataStream operation, every `query` call would instantiate a
> new, independent
> service. This makes it difficult to share a single service across multiple
> operators within
> the same job, which is a key use case we want to support.
>
>
> With the `addRpcOperatorService` / `startSharedRpcOperatorService` pair,
> both per-job
> and shared scenarios are covered through a unified and explicit API. Do
> you think this
> approach works?
>
>
> If you are aligned with this approach, I will update the FLIP accordingly.
> Thank you again
> for raising these important points!
>
>
> Best,
> Yi
>
>
>
> At 2026-06-02 14:27:26, "Guowei Ma" <gu...@gmail.com> wrote:
> >Hi, YI
> >Thanks a lot for the explanation and the update. I still have two small
> >questions I'd like to get your thoughts on:
> >
> >*1. On auto-scaling*
> >
> >I think auto-scaling should be a first-class citizen in Flink as a whole —
> >and RpcOperator was proposed precisely to make this achievable for GPU
> >workloads. For that reason, baking a fixed parallelism into the API as the
> >first step doesn't feel right to me; put simply, I don't think we should
> >introduce an API that locks parallelism to a single value.
> >
> >That said, I fully agree that a complete auto-scaling implementation is
> >complex and isn't the focus of this FLIP. So my suggestion is that the
> >first version could just support the degenerate case — e.g. min = max —
> >without committing the API itself to fixed parallelism.
> >
> >*2. On the lifecycle question*
> >
> >a. This is actually tied to how we express the API. What I've had in mind
> >is something like ds.query(new RPCOperatorService), since it leaves no
> room
> >for ambiguity: the service clearly belongs to a single job, and every job
> >has its own.
> >
> >b. Consider this scenario: a Flink application-mode deployment containing
> >two jobs, where each job needs its own independent RPC service. With your
> >current design, would I have to call addRpcService twice?
> >
> >c. If it's StreamExecutionEnvironment that interacts with the RPC
> operator,
> >that feels more like something shared *across* jobs — analogous to
> >registering a UDF in the Table API, where multiple jobs can use it. That
> >seems to sit in tension with the per-job semantics above.
> >Best,
> >Guowei
> >
> >
> >On Tue, Jun 2, 2026 at 11:16 AM Gen Luo <lu...@gmail.com> wrote:
> >
> >> Hi Yi,
> >>
> >> Thanks for this impressive proposal. Adding an RPC service deployment
> >> primitive can greatly expand Flink's capability scope and flexibility,
> with
> >> clear benefits for AI workloads and agent applications. It can also
> play a
> >> role in scenarios like state management or data lake compaction —
> relevant
> >> to but decoupled from the data pipeline. This is a very promising
> >> direction. That said, I still have some questions.
> >>
> >> 1. My biggest concern is heterogeneous resources, echoing Hongshun's Q4.
> >> Flink today uses homogeneous TM resources sliced for different task
> >> requirements, but decoupling GPU/CPU resources and enabling flexible
> >> independent scaling is one of the core values of this FLIP — and the
> >> current resource allocation approach cannot support this. We likely need
> >> new implementations for the resource allocation component.
> >>
> >> 2. Some components are described only at the idea level without
> >> implementation details. Hopefully these can be supplemented later or
> >> discussed in dedicated threads. A few worth discussing:
> >>
> >> - RpcTask: It's currently described as a simple single-threaded loop,
> >> with no plan to use Mailbox or OperatorChain. But if we want to support
> >> async, checkpoint, etc. later, introducing Mailbox would be valuable —
> >> handle the input queue as defaultAction, with async/CP/system messages
> as
> >> mails. This keeps semantics consistent with StreamTask and reduces
> >> maintenance burden. System events like FO-triggered request cancellation
> >> can also be prioritized via mailbox. OperatorChain is what
> >> SubtaskCheckpointCoordinator directly contacts, so having RpcOperator as
> >> the only node in the chain wouldn't add much complexity and lowers the
> cost
> >> when CP support comes.
> >>
> >> - Communication implementation: Will it reuse the Pekko + proxy
> >> mechanism? Pekko has risks with large message bodies (OOM, silently drop
> >> oversized frame, etc.). Since ROS is user-facing, we can't fully avoid
> >> this, and large-model scenarios often carry sizable context — needs
> careful
> >> handling. Also, the current design doesn't include table-function-style
> >> interfaces for unfolding data, so a large unfolded result has to be
> >> returned as a single response, inflating per-request payload size and
> >> amplifying the same risk.
> >>
> >> - Client-side implementation: Only the interface is described, no
> >> implementation details. Checkpoint is the prominent issue — since ROS is
> >> stateless by design, state recording falls to the client. Do we go with
> >> drain or record? Drain may take a long time; record on a keyed stream
> may
> >> cause request distribution confusion after parallelism changes.
> >>
> >> 3. ROS as container, not service: Is this allowed? If the system itself
> has
> >> service capabilities — especially non-Java stacks (agents, LLMs, also
> >> Python remote functions) — forwarding requests through the Flink Java
> layer
> >> adds unnecessary overhead. In that case RpcOperator would only handle
> >> lifecycle, while the framework manages resource scheduling and service
> >> discovery without carrying business traffic.
> >>
> >> nit: addRpcService is inconsistent with other places that use
> RpcOperator /
> >> the ros keyword. Worth unifying.
> >>
> >> Best,
> >> Gen
> >>
> >> On Mon, Jun 1, 2026 at 2:02 PM Yi Zhang <zh...@163.com> wrote:
> >>
> >> > Hi Guowei,
> >> >
> >> >
> >> > Thanks for the thoughtful review and valuable feedback! Your questions
> >> are
> >> > insightful
> >> > and have directly driven refinements to the proposal. Here are my
> >> > responses to each
> >> > point:
> >> >
> >> >
> >> > 1. Autoscaling
> >> > Supporting a (min, max) concurrency range is a great idea. However,
> >> > auto-scaling
> >> > involves much more than just the API surface — it requires discussion
> >> > around
> >> > scaling policies, coordination with the scheduler, metric-driven
> >> triggers,
> >> > and more.
> >> > To keep this FLIP focused, I'd like to defer that to a follow-up.
> Fixed
> >> > parallelism is
> >> > a common and necessary capability on its own, and the first version
> will
> >> > support
> >> > that. The interfaces can be extended later to include auto-scaling
> >> support
> >> > without
> >> > breaking changes.
> >> >
> >> >
> >> > 2. Async invocation & end-to-end examples
> >> > This is a good point. The `RosClient` already provides an async
> >> interface,
> >> > so using
> >> > it with `AsyncDataStream` is indeed more natural and idiomatic. I will
> >> add
> >> > an
> >> > asynchronous invocation example in the proposal. Regarding a complete
> >> > end-to-end
> >> > inference example: at the Java API level, it will still be primarily
> >> > conceptual, since
> >> > practical model inference examples really need Python API support
> (e.g.,
> >> > loading a
> >> > PyTorch or ONNX model). I will refine the existing example to make the
> >> > overall
> >> > invocation chain as close to a real-world scenario as possible.
> >> >
> >> >
> >> > 3. GPU as a resource
> >> > This is indeed a key concern. The `extendedResource` API in the
> original
> >> > proposal
> >> > comes from Flink's existing resource model (introduced by FLIP-108).
> >> > However, I
> >> > agree that GPU deserves first-class treatment rather than being
> accessed
> >> > through a
> >> > generic extension mechanism. Promoting GPU to a first-class resource
> will
> >> > involve
> >> > changes to both the API and the scheduling layer, which we plan to
> >> address
> >> > in a
> >> > dedicated FLIP. For now, I will remove the `extendedResource` API from
> >> > `RosDescriptor` to keep this FLIP focused on the RPC service primitive
> >> > itself. The
> >> > resource declaration API can be extended once the GPU-specific FLIP
> is in
> >> > place.
> >> >
> >> >
> >> > 4. Serial method invocation
> >> > Agreed. I will make the serial invocation semantics more prominent —
> both
> >> > in the
> >> > FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
> >> >
> >> >
> >> > 5. Why synchronous methods?
> >> > This is a key question. The initial design chose synchronous methods
> for
> >> > simplicity
> >> > — users don't need to worry about thread safety, and concurrency is
> >> > achieved by
> >> > scaling out instances (parallelism). However, I recognize that
> >> > asynchronous
> >> > processing is a critical requirement for certain scenarios (e.g., when
> >> the
> >> > RPC service
> >> > performs expensive operations that can be parallelized internally). I
> am
> >> > planning to
> >> > extend the current design to support asynchronous method execution:
> if a
> >> > user's
> >> > method returns a `CompletableFuture`, the framework will recognize it
> and
> >> > wait for
> >> > the future to complete before sending the response, without blocking
> the
> >> > invoke
> >> > thread. This allows the invoke thread to continue dispatching
> subsequent
> >> > requests,
> >> > while the user controls internal concurrency. Importantly,
> >> > `processRequest` calls
> >> > remain serial — the async behavior is purely opt-in.
> >> >
> >> >
> >> > 6. SQL UDF usage
> >> > The current design is primarily oriented toward the DataStream API,
> as it
> >> > provides
> >> > the foundational building blocks. I agree that SQL and Table API
> support
> >> > is an
> >> > important aspect to consider as well. On the invocation side, both
> Table
> >> > API and
> >> > SQL UDFs share the same runtime path through `FunctionContext`, so
> >> > enabling
> >> > RPC service access from UDFs is straightforward — I will add a
> >> > `getRosClient`
> >> > method to `FunctionContext` and include that change in this FLIP.
> >> > The real challenge for SQL lies on the definition side: how to define
> and
> >> > register
> >> > an RPC service using pure SQL syntax, which would require SQL grammar
> >> > extensions and planner changes. I think that part would be better
> >> > addressed in
> >> > a dedicated follow-up FLIP to give it the attention it deserves.
> >> >
> >> >
> >> > 7. Service sharing across jobs & lifecycle
> >> > In this FLIP, the RPC service is defined as part of the job (via
> >> > `env.addRpcService()`
> >> > before `env.execute()`). It cannot be accessed across jobs, and it is
> >> > released when
> >> > the job reaches a terminal state. I believe your question is more
> >> relevant
> >> > to the
> >> > "Shared RPC Service" mentioned in the Follow-up Tasks section. While
> it's
> >> > out of
> >> > scope for this FLIP, let me briefly describe the envisioned design: a
> >> > shared RPC
> >> > service would run as an independent job within the same Application,
> >> > accessible by
> >> > all jobs in that Application. It would terminate when the Application
> >> > ends, or could be
> >> > stopped manually via a user-facing API.
> >> > The embedded (per-job) service model discussed in this FLIP is
> simpler to
> >> > manage
> >> > and well-suited for many streaming scenarios, which is why we focus
> on it
> >> > first.
> >> > Importantly, both the embedded and shared modes will reuse the same
> set
> >> of
> >> > deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`,
> etc.)
> >> > — the
> >> > difference is only in lifecycle management and visibility scope.
> >> >
> >> >
> >> >
> >> >
> >> > I hope the above addresses your questions. I will update the proposal
> to
> >> > reflect the
> >> > changes mentioned above. Please feel free to follow up if you have any
> >> > further
> >> > questions or suggestions.
> >> >
> >> >
> >> > Best,
> >> > Yi
> >> >
> >> >
> >> >
> >> > At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
> >> > >HI,YI
> >> > >
> >> > >One more question I'd like to raise is around how the service behaves
> >> when
> >> > >it's referenced by more than one job. If two jobs are submitted
> within
> >> the
> >> > >same application and both reference the same RpcOperator service, is
> >> that
> >> > >service shared across them, and what is the expected behavior in that
> >> > case?
> >> > >I'm also curious how this plays out for bounded (batch) jobs in
> >> > particular:
> >> > >if one job completes while the other is still running, what does the
> >> > >service lifecycle look like — is the service torn down as soon as the
> >> > first
> >> > >job finishes, and if so, how does that affect the job that's still
> >> > running?
> >> > >
> >> > >Best,
> >> > >Guowei
> >> > >
> >> > >
> >> > >On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com>
> wrote:
> >> > >
> >> > >> Hi,YI
> >> > >>
> >> > >> Thanks for putting together this FLIP — it's a well-thought-out
> >> > proposal,
> >> > >> and the RpcOperator Service is a really valuable addition for
> >> > AI/inference
> >> > >> workloads on Flink. I have a few questions and suggestions:
> >> > >>
> >> > >>
> >> > >> 1. Autoscaling
> >> > >> How is scale-up/scale-down expressed in this design? Could you
> >> consider
> >> > >> allowing concurrency to be configured as a (min, max) range?
> Currently
> >> > it
> >> > >> appears only min = max (i.e. fixed parallelism) is supported.
> >> > >>
> >> > >>
> >> > >> 2. Async invocation & end-to-end examples
> >> > >> The example shows a synchronous invocation style. I'd suggest also
> >> > >> providing an asynchronous invocation example — or even just a
> scaffold
> >> > that
> >> > >> exposes only the RpcOperator's handle and method. In addition, it
> >> would
> >> > be
> >> > >> best to include a complete end-to-end example of invoking a model
> (CPU
> >> > >> inference would be fine).
> >> > >>
> >> > >>
> >> > >> 3. GPU as a resource
> >> > >> GPU is surely a common/first-class resource. Accessing it here
> through
> >> > an
> >> > >> extend mechanism — wouldn't that mislead users? It would seem more
> >> > >> natural to declare GPU as a standard resource rather than going
> >> through
> >> > an
> >> > >> extension.
> >> > >>
> >> > >>
> >> > >> 4. Serial method invocation
> >> > >> Each method of an RpcOperator is invoked serially. I'd suggest
> stating
> >> > >> this explicitly in both the documentation and the code comments, so
> >> that
> >> > >> users — or an AI — can be clearly aware of it.
> >> > >>
> >> > >>
> >> > >> 5. Why synchronous methods?
> >> > >> Why is every method in RpcOperator a synchronous call? Does this
> imply
> >> > >> that no model inference supports concurrent invocation?
> >> > >>
> >> > >>
> >> > >> 6. SQL UDF usage
> >> > >> How would SQL UDF users invoke this capability? I'd suggest
> providing
> >> a
> >> > >> complete example for this as well.
> >> > >>
> >> > >>
> >> > >> Thanks again for the great work!
> >> > >>
> >> > >> Best,
> >> > >> Guowei
> >> > >>
> >> > >>
> >> > >> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <
> dockerzhang@apache.org
> >> >
> >> > >> wrote:
> >> > >>
> >> > >>> Hi Yi, and Flink Community,
> >> > >>>
> >> > >>> Thanks for bringing up this excellent proposal. I am fully in
> favor
> >> of
> >> > >>> FLIP-582.
> >> > >>>
> >> > >>> In our production workloads, especially regarding large-scale AI
> >> > >>> inference,
> >> > >>> the tight coupling of the data plane and compute units has always
> >> been
> >> > a
> >> > >>> major pain point. If an inference subtask fails due to a GPU OOM
> or
> >> > driver
> >> > >>> issue, triggering a global rollback is incredibly expensive since
> >> model
> >> > >>> reloading takes minutes.
> >> > >>>
> >> > >>> The introduction of the RpcOperator Service as a first-class
> >> primitive
> >> > >>> masterfully decouples the heavy inference tasks from the
> mainstream
> >> > >>> topology. The fault isolation, independent scaling, and stateless
> >> > design
> >> > >>> perfectly match the requirements of modern AI-oriented data
> >> processing.
> >> > >>>
> >> > >>> This is a clean and robust architecture. Looking forward to seeing
> >> this
> >> > >>> merged!
> >> > >>>
> >> > >>>
> >> > >>> Best wishes,
> >> > >>> Charles Zhang
> >> > >>> from Apache InLong
> >> > >>>
> >> > >>>
> >> > >>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
> >> > >>>
> >> > >>> > Hi everyone,
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > I would like to start a discussion on FLIP-582: Support
> RpcOperator
> >> > >>> > Service [1].
> >> > >>> >
> >> > >>> >
> >> > >>> > AI-oriented workloads like multimodal data processing and model
> >> > >>> inference
> >> > >>> > are
> >> > >>> > growing rapidly in recent years. These workloads are
> characterized
> >> by
> >> > >>> > expensive
> >> > >>> > resources (GPUs) and high initialization costs (seconds to
> minutes
> >> > for
> >> > >>> > model
> >> > >>> > loading). In today's Flink, embedding them in the data plane
> >> couples
> >> > >>> their
> >> > >>> > parallelism and failover with surrounding operators; deploying
> them
> >> > as
> >> > >>> > external
> >> > >>> > services disconnects their lifecycle from the job and doubles
> >> > >>> operational
> >> > >>> > overhead.
> >> > >>> >
> >> > >>> >
> >> > >>> > This FLIP introduces RpcOperator Service — a framework-level
> >> > primitive
> >> > >>> > that runs
> >> > >>> > user-defined compute as RPC services in an independent Pipelined
> >> > Region
> >> > >>> > within
> >> > >>> > the Flink job. Because the service is isolated at the scheduling
> >> > level,
> >> > >>> it
> >> > >>> > can achieve
> >> > >>> > fault isolation, independent scaling, and dedicated resource
> >> > allocation.
> >> > >>> > As a native
> >> > >>> > Flink primitive, it also lays the foundation for automatic flow
> >> > control,
> >> > >>> > flexible load
> >> > >>> > balancing, and coordinated auto-scaling — all without
> introducing
> >> > >>> external
> >> > >>> > infrastructure or additional operational burden.
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > Looking forward to your feedback and suggestions!
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > [1]
> >> > >>> >
> >> > >>> >
> >> > >>>
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > Best Regards,
> >> > >>> > Yi Zhang
> >> > >>>
> >> > >>
> >> >
> >>
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Guowei,
Thank you for your patience — I'm sorry for the delayed reply. I really appreciate your
follow-up questions; they point to two key design choices that deserve thorough
consideration.
1. Auto-scaling & parallelism API
I believe we are in agreement that auto-scaling logic will not be included in this FLIP.
Given that, I wonder whether introducing `min` and `max` parameters at this stage
might actually confuse users — they would see an API that accepts a range but offers
no runtime behavior to act on it.
I'd like to clarify how I think about the `parallelism` parameter: it does not necessarily
represent a "fixed" parallelism, but rather an **initial parallelism** — the service starts
with this level of concurrency and may be adjusted during subsequent execution. Of
course, such adjustment would indeed require bounds. Would it be acceptable to
introduce them when we actually bring in the auto-scaling capability? This way, the
initial API remains simple and intuitive.
2. Lifecycle semantics and API expression
This is a great question, and it actually connects closely to the **Shared RPC Service**
task I mentioned in the Follow-up section. After further discussion and reflection, I now
believe it would be better to include the shared service capability in this FLIP as well.
The idea is to introduce a separate API: `env.startSharedRpcOperatorService(...)`, which
is clearly distinct from `env.addRpcOperatorService(...)`. The semantics would be:
a. `addRpcOperatorService` retains its **per-job** semantics, similar to how
transformations are cleared after `execute()`, services added via this API are also scoped
to a single job execution and cleaned up afterward.
b. `startSharedRpcOperatorService` starts a service that lives independently of any
particular job, accessible by multiple jobs within the same application.
Regarding your suggestion of `ds.query(new RPCOperatorService)` — it's a nice idea
for expressing per-job ownership, but I see one limitation: since the service is bound to
a specific DataStream operation, every `query` call would instantiate a new, independent
service. This makes it difficult to share a single service across multiple operators within
the same job, which is a key use case we want to support.
With the `addRpcOperatorService` / `startSharedRpcOperatorService` pair, both per-job
and shared scenarios are covered through a unified and explicit API. Do you think this
approach works?
If you are aligned with this approach, I will update the FLIP accordingly. Thank you again
for raising these important points!
Best,
Yi
At 2026-06-02 14:27:26, "Guowei Ma" <gu...@gmail.com> wrote:
>Hi, YI
>Thanks a lot for the explanation and the update. I still have two small
>questions I'd like to get your thoughts on:
>
>*1. On auto-scaling*
>
>I think auto-scaling should be a first-class citizen in Flink as a whole —
>and RpcOperator was proposed precisely to make this achievable for GPU
>workloads. For that reason, baking a fixed parallelism into the API as the
>first step doesn't feel right to me; put simply, I don't think we should
>introduce an API that locks parallelism to a single value.
>
>That said, I fully agree that a complete auto-scaling implementation is
>complex and isn't the focus of this FLIP. So my suggestion is that the
>first version could just support the degenerate case — e.g. min = max —
>without committing the API itself to fixed parallelism.
>
>*2. On the lifecycle question*
>
>a. This is actually tied to how we express the API. What I've had in mind
>is something like ds.query(new RPCOperatorService), since it leaves no room
>for ambiguity: the service clearly belongs to a single job, and every job
>has its own.
>
>b. Consider this scenario: a Flink application-mode deployment containing
>two jobs, where each job needs its own independent RPC service. With your
>current design, would I have to call addRpcService twice?
>
>c. If it's StreamExecutionEnvironment that interacts with the RPC operator,
>that feels more like something shared *across* jobs — analogous to
>registering a UDF in the Table API, where multiple jobs can use it. That
>seems to sit in tension with the per-job semantics above.
>Best,
>Guowei
>
>
>On Tue, Jun 2, 2026 at 11:16 AM Gen Luo <lu...@gmail.com> wrote:
>
>> Hi Yi,
>>
>> Thanks for this impressive proposal. Adding an RPC service deployment
>> primitive can greatly expand Flink's capability scope and flexibility, with
>> clear benefits for AI workloads and agent applications. It can also play a
>> role in scenarios like state management or data lake compaction — relevant
>> to but decoupled from the data pipeline. This is a very promising
>> direction. That said, I still have some questions.
>>
>> 1. My biggest concern is heterogeneous resources, echoing Hongshun's Q4.
>> Flink today uses homogeneous TM resources sliced for different task
>> requirements, but decoupling GPU/CPU resources and enabling flexible
>> independent scaling is one of the core values of this FLIP — and the
>> current resource allocation approach cannot support this. We likely need
>> new implementations for the resource allocation component.
>>
>> 2. Some components are described only at the idea level without
>> implementation details. Hopefully these can be supplemented later or
>> discussed in dedicated threads. A few worth discussing:
>>
>> - RpcTask: It's currently described as a simple single-threaded loop,
>> with no plan to use Mailbox or OperatorChain. But if we want to support
>> async, checkpoint, etc. later, introducing Mailbox would be valuable —
>> handle the input queue as defaultAction, with async/CP/system messages as
>> mails. This keeps semantics consistent with StreamTask and reduces
>> maintenance burden. System events like FO-triggered request cancellation
>> can also be prioritized via mailbox. OperatorChain is what
>> SubtaskCheckpointCoordinator directly contacts, so having RpcOperator as
>> the only node in the chain wouldn't add much complexity and lowers the cost
>> when CP support comes.
>>
>> - Communication implementation: Will it reuse the Pekko + proxy
>> mechanism? Pekko has risks with large message bodies (OOM, silently drop
>> oversized frame, etc.). Since ROS is user-facing, we can't fully avoid
>> this, and large-model scenarios often carry sizable context — needs careful
>> handling. Also, the current design doesn't include table-function-style
>> interfaces for unfolding data, so a large unfolded result has to be
>> returned as a single response, inflating per-request payload size and
>> amplifying the same risk.
>>
>> - Client-side implementation: Only the interface is described, no
>> implementation details. Checkpoint is the prominent issue — since ROS is
>> stateless by design, state recording falls to the client. Do we go with
>> drain or record? Drain may take a long time; record on a keyed stream may
>> cause request distribution confusion after parallelism changes.
>>
>> 3. ROS as container, not service: Is this allowed? If the system itself has
>> service capabilities — especially non-Java stacks (agents, LLMs, also
>> Python remote functions) — forwarding requests through the Flink Java layer
>> adds unnecessary overhead. In that case RpcOperator would only handle
>> lifecycle, while the framework manages resource scheduling and service
>> discovery without carrying business traffic.
>>
>> nit: addRpcService is inconsistent with other places that use RpcOperator /
>> the ros keyword. Worth unifying.
>>
>> Best,
>> Gen
>>
>> On Mon, Jun 1, 2026 at 2:02 PM Yi Zhang <zh...@163.com> wrote:
>>
>> > Hi Guowei,
>> >
>> >
>> > Thanks for the thoughtful review and valuable feedback! Your questions
>> are
>> > insightful
>> > and have directly driven refinements to the proposal. Here are my
>> > responses to each
>> > point:
>> >
>> >
>> > 1. Autoscaling
>> > Supporting a (min, max) concurrency range is a great idea. However,
>> > auto-scaling
>> > involves much more than just the API surface — it requires discussion
>> > around
>> > scaling policies, coordination with the scheduler, metric-driven
>> triggers,
>> > and more.
>> > To keep this FLIP focused, I'd like to defer that to a follow-up. Fixed
>> > parallelism is
>> > a common and necessary capability on its own, and the first version will
>> > support
>> > that. The interfaces can be extended later to include auto-scaling
>> support
>> > without
>> > breaking changes.
>> >
>> >
>> > 2. Async invocation & end-to-end examples
>> > This is a good point. The `RosClient` already provides an async
>> interface,
>> > so using
>> > it with `AsyncDataStream` is indeed more natural and idiomatic. I will
>> add
>> > an
>> > asynchronous invocation example in the proposal. Regarding a complete
>> > end-to-end
>> > inference example: at the Java API level, it will still be primarily
>> > conceptual, since
>> > practical model inference examples really need Python API support (e.g.,
>> > loading a
>> > PyTorch or ONNX model). I will refine the existing example to make the
>> > overall
>> > invocation chain as close to a real-world scenario as possible.
>> >
>> >
>> > 3. GPU as a resource
>> > This is indeed a key concern. The `extendedResource` API in the original
>> > proposal
>> > comes from Flink's existing resource model (introduced by FLIP-108).
>> > However, I
>> > agree that GPU deserves first-class treatment rather than being accessed
>> > through a
>> > generic extension mechanism. Promoting GPU to a first-class resource will
>> > involve
>> > changes to both the API and the scheduling layer, which we plan to
>> address
>> > in a
>> > dedicated FLIP. For now, I will remove the `extendedResource` API from
>> > `RosDescriptor` to keep this FLIP focused on the RPC service primitive
>> > itself. The
>> > resource declaration API can be extended once the GPU-specific FLIP is in
>> > place.
>> >
>> >
>> > 4. Serial method invocation
>> > Agreed. I will make the serial invocation semantics more prominent — both
>> > in the
>> > FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
>> >
>> >
>> > 5. Why synchronous methods?
>> > This is a key question. The initial design chose synchronous methods for
>> > simplicity
>> > — users don't need to worry about thread safety, and concurrency is
>> > achieved by
>> > scaling out instances (parallelism). However, I recognize that
>> > asynchronous
>> > processing is a critical requirement for certain scenarios (e.g., when
>> the
>> > RPC service
>> > performs expensive operations that can be parallelized internally). I am
>> > planning to
>> > extend the current design to support asynchronous method execution: if a
>> > user's
>> > method returns a `CompletableFuture`, the framework will recognize it and
>> > wait for
>> > the future to complete before sending the response, without blocking the
>> > invoke
>> > thread. This allows the invoke thread to continue dispatching subsequent
>> > requests,
>> > while the user controls internal concurrency. Importantly,
>> > `processRequest` calls
>> > remain serial — the async behavior is purely opt-in.
>> >
>> >
>> > 6. SQL UDF usage
>> > The current design is primarily oriented toward the DataStream API, as it
>> > provides
>> > the foundational building blocks. I agree that SQL and Table API support
>> > is an
>> > important aspect to consider as well. On the invocation side, both Table
>> > API and
>> > SQL UDFs share the same runtime path through `FunctionContext`, so
>> > enabling
>> > RPC service access from UDFs is straightforward — I will add a
>> > `getRosClient`
>> > method to `FunctionContext` and include that change in this FLIP.
>> > The real challenge for SQL lies on the definition side: how to define and
>> > register
>> > an RPC service using pure SQL syntax, which would require SQL grammar
>> > extensions and planner changes. I think that part would be better
>> > addressed in
>> > a dedicated follow-up FLIP to give it the attention it deserves.
>> >
>> >
>> > 7. Service sharing across jobs & lifecycle
>> > In this FLIP, the RPC service is defined as part of the job (via
>> > `env.addRpcService()`
>> > before `env.execute()`). It cannot be accessed across jobs, and it is
>> > released when
>> > the job reaches a terminal state. I believe your question is more
>> relevant
>> > to the
>> > "Shared RPC Service" mentioned in the Follow-up Tasks section. While it's
>> > out of
>> > scope for this FLIP, let me briefly describe the envisioned design: a
>> > shared RPC
>> > service would run as an independent job within the same Application,
>> > accessible by
>> > all jobs in that Application. It would terminate when the Application
>> > ends, or could be
>> > stopped manually via a user-facing API.
>> > The embedded (per-job) service model discussed in this FLIP is simpler to
>> > manage
>> > and well-suited for many streaming scenarios, which is why we focus on it
>> > first.
>> > Importantly, both the embedded and shared modes will reuse the same set
>> of
>> > deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`, etc.)
>> > — the
>> > difference is only in lifecycle management and visibility scope.
>> >
>> >
>> >
>> >
>> > I hope the above addresses your questions. I will update the proposal to
>> > reflect the
>> > changes mentioned above. Please feel free to follow up if you have any
>> > further
>> > questions or suggestions.
>> >
>> >
>> > Best,
>> > Yi
>> >
>> >
>> >
>> > At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
>> > >HI,YI
>> > >
>> > >One more question I'd like to raise is around how the service behaves
>> when
>> > >it's referenced by more than one job. If two jobs are submitted within
>> the
>> > >same application and both reference the same RpcOperator service, is
>> that
>> > >service shared across them, and what is the expected behavior in that
>> > case?
>> > >I'm also curious how this plays out for bounded (batch) jobs in
>> > particular:
>> > >if one job completes while the other is still running, what does the
>> > >service lifecycle look like — is the service torn down as soon as the
>> > first
>> > >job finishes, and if so, how does that affect the job that's still
>> > running?
>> > >
>> > >Best,
>> > >Guowei
>> > >
>> > >
>> > >On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
>> > >
>> > >> Hi,YI
>> > >>
>> > >> Thanks for putting together this FLIP — it's a well-thought-out
>> > proposal,
>> > >> and the RpcOperator Service is a really valuable addition for
>> > AI/inference
>> > >> workloads on Flink. I have a few questions and suggestions:
>> > >>
>> > >>
>> > >> 1. Autoscaling
>> > >> How is scale-up/scale-down expressed in this design? Could you
>> consider
>> > >> allowing concurrency to be configured as a (min, max) range? Currently
>> > it
>> > >> appears only min = max (i.e. fixed parallelism) is supported.
>> > >>
>> > >>
>> > >> 2. Async invocation & end-to-end examples
>> > >> The example shows a synchronous invocation style. I'd suggest also
>> > >> providing an asynchronous invocation example — or even just a scaffold
>> > that
>> > >> exposes only the RpcOperator's handle and method. In addition, it
>> would
>> > be
>> > >> best to include a complete end-to-end example of invoking a model (CPU
>> > >> inference would be fine).
>> > >>
>> > >>
>> > >> 3. GPU as a resource
>> > >> GPU is surely a common/first-class resource. Accessing it here through
>> > an
>> > >> extend mechanism — wouldn't that mislead users? It would seem more
>> > >> natural to declare GPU as a standard resource rather than going
>> through
>> > an
>> > >> extension.
>> > >>
>> > >>
>> > >> 4. Serial method invocation
>> > >> Each method of an RpcOperator is invoked serially. I'd suggest stating
>> > >> this explicitly in both the documentation and the code comments, so
>> that
>> > >> users — or an AI — can be clearly aware of it.
>> > >>
>> > >>
>> > >> 5. Why synchronous methods?
>> > >> Why is every method in RpcOperator a synchronous call? Does this imply
>> > >> that no model inference supports concurrent invocation?
>> > >>
>> > >>
>> > >> 6. SQL UDF usage
>> > >> How would SQL UDF users invoke this capability? I'd suggest providing
>> a
>> > >> complete example for this as well.
>> > >>
>> > >>
>> > >> Thanks again for the great work!
>> > >>
>> > >> Best,
>> > >> Guowei
>> > >>
>> > >>
>> > >> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <dockerzhang@apache.org
>> >
>> > >> wrote:
>> > >>
>> > >>> Hi Yi, and Flink Community,
>> > >>>
>> > >>> Thanks for bringing up this excellent proposal. I am fully in favor
>> of
>> > >>> FLIP-582.
>> > >>>
>> > >>> In our production workloads, especially regarding large-scale AI
>> > >>> inference,
>> > >>> the tight coupling of the data plane and compute units has always
>> been
>> > a
>> > >>> major pain point. If an inference subtask fails due to a GPU OOM or
>> > driver
>> > >>> issue, triggering a global rollback is incredibly expensive since
>> model
>> > >>> reloading takes minutes.
>> > >>>
>> > >>> The introduction of the RpcOperator Service as a first-class
>> primitive
>> > >>> masterfully decouples the heavy inference tasks from the mainstream
>> > >>> topology. The fault isolation, independent scaling, and stateless
>> > design
>> > >>> perfectly match the requirements of modern AI-oriented data
>> processing.
>> > >>>
>> > >>> This is a clean and robust architecture. Looking forward to seeing
>> this
>> > >>> merged!
>> > >>>
>> > >>>
>> > >>> Best wishes,
>> > >>> Charles Zhang
>> > >>> from Apache InLong
>> > >>>
>> > >>>
>> > >>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>> > >>>
>> > >>> > Hi everyone,
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > I would like to start a discussion on FLIP-582: Support RpcOperator
>> > >>> > Service [1].
>> > >>> >
>> > >>> >
>> > >>> > AI-oriented workloads like multimodal data processing and model
>> > >>> inference
>> > >>> > are
>> > >>> > growing rapidly in recent years. These workloads are characterized
>> by
>> > >>> > expensive
>> > >>> > resources (GPUs) and high initialization costs (seconds to minutes
>> > for
>> > >>> > model
>> > >>> > loading). In today's Flink, embedding them in the data plane
>> couples
>> > >>> their
>> > >>> > parallelism and failover with surrounding operators; deploying them
>> > as
>> > >>> > external
>> > >>> > services disconnects their lifecycle from the job and doubles
>> > >>> operational
>> > >>> > overhead.
>> > >>> >
>> > >>> >
>> > >>> > This FLIP introduces RpcOperator Service — a framework-level
>> > primitive
>> > >>> > that runs
>> > >>> > user-defined compute as RPC services in an independent Pipelined
>> > Region
>> > >>> > within
>> > >>> > the Flink job. Because the service is isolated at the scheduling
>> > level,
>> > >>> it
>> > >>> > can achieve
>> > >>> > fault isolation, independent scaling, and dedicated resource
>> > allocation.
>> > >>> > As a native
>> > >>> > Flink primitive, it also lays the foundation for automatic flow
>> > control,
>> > >>> > flexible load
>> > >>> > balancing, and coordinated auto-scaling — all without introducing
>> > >>> external
>> > >>> > infrastructure or additional operational burden.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > Looking forward to your feedback and suggestions!
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > [1]
>> > >>> >
>> > >>> >
>> > >>>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > Best Regards,
>> > >>> > Yi Zhang
>> > >>>
>> > >>
>> >
>>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Guowei Ma <gu...@gmail.com>.
Hi, YI
Thanks a lot for the explanation and the update. I still have two small
questions I'd like to get your thoughts on:
*1. On auto-scaling*
I think auto-scaling should be a first-class citizen in Flink as a whole —
and RpcOperator was proposed precisely to make this achievable for GPU
workloads. For that reason, baking a fixed parallelism into the API as the
first step doesn't feel right to me; put simply, I don't think we should
introduce an API that locks parallelism to a single value.
That said, I fully agree that a complete auto-scaling implementation is
complex and isn't the focus of this FLIP. So my suggestion is that the
first version could just support the degenerate case — e.g. min = max —
without committing the API itself to fixed parallelism.
*2. On the lifecycle question*
a. This is actually tied to how we express the API. What I've had in mind
is something like ds.query(new RPCOperatorService), since it leaves no room
for ambiguity: the service clearly belongs to a single job, and every job
has its own.
b. Consider this scenario: a Flink application-mode deployment containing
two jobs, where each job needs its own independent RPC service. With your
current design, would I have to call addRpcService twice?
c. If it's StreamExecutionEnvironment that interacts with the RPC operator,
that feels more like something shared *across* jobs — analogous to
registering a UDF in the Table API, where multiple jobs can use it. That
seems to sit in tension with the per-job semantics above.
Best,
Guowei
On Tue, Jun 2, 2026 at 11:16 AM Gen Luo <lu...@gmail.com> wrote:
> Hi Yi,
>
> Thanks for this impressive proposal. Adding an RPC service deployment
> primitive can greatly expand Flink's capability scope and flexibility, with
> clear benefits for AI workloads and agent applications. It can also play a
> role in scenarios like state management or data lake compaction — relevant
> to but decoupled from the data pipeline. This is a very promising
> direction. That said, I still have some questions.
>
> 1. My biggest concern is heterogeneous resources, echoing Hongshun's Q4.
> Flink today uses homogeneous TM resources sliced for different task
> requirements, but decoupling GPU/CPU resources and enabling flexible
> independent scaling is one of the core values of this FLIP — and the
> current resource allocation approach cannot support this. We likely need
> new implementations for the resource allocation component.
>
> 2. Some components are described only at the idea level without
> implementation details. Hopefully these can be supplemented later or
> discussed in dedicated threads. A few worth discussing:
>
> - RpcTask: It's currently described as a simple single-threaded loop,
> with no plan to use Mailbox or OperatorChain. But if we want to support
> async, checkpoint, etc. later, introducing Mailbox would be valuable —
> handle the input queue as defaultAction, with async/CP/system messages as
> mails. This keeps semantics consistent with StreamTask and reduces
> maintenance burden. System events like FO-triggered request cancellation
> can also be prioritized via mailbox. OperatorChain is what
> SubtaskCheckpointCoordinator directly contacts, so having RpcOperator as
> the only node in the chain wouldn't add much complexity and lowers the cost
> when CP support comes.
>
> - Communication implementation: Will it reuse the Pekko + proxy
> mechanism? Pekko has risks with large message bodies (OOM, silently drop
> oversized frame, etc.). Since ROS is user-facing, we can't fully avoid
> this, and large-model scenarios often carry sizable context — needs careful
> handling. Also, the current design doesn't include table-function-style
> interfaces for unfolding data, so a large unfolded result has to be
> returned as a single response, inflating per-request payload size and
> amplifying the same risk.
>
> - Client-side implementation: Only the interface is described, no
> implementation details. Checkpoint is the prominent issue — since ROS is
> stateless by design, state recording falls to the client. Do we go with
> drain or record? Drain may take a long time; record on a keyed stream may
> cause request distribution confusion after parallelism changes.
>
> 3. ROS as container, not service: Is this allowed? If the system itself has
> service capabilities — especially non-Java stacks (agents, LLMs, also
> Python remote functions) — forwarding requests through the Flink Java layer
> adds unnecessary overhead. In that case RpcOperator would only handle
> lifecycle, while the framework manages resource scheduling and service
> discovery without carrying business traffic.
>
> nit: addRpcService is inconsistent with other places that use RpcOperator /
> the ros keyword. Worth unifying.
>
> Best,
> Gen
>
> On Mon, Jun 1, 2026 at 2:02 PM Yi Zhang <zh...@163.com> wrote:
>
> > Hi Guowei,
> >
> >
> > Thanks for the thoughtful review and valuable feedback! Your questions
> are
> > insightful
> > and have directly driven refinements to the proposal. Here are my
> > responses to each
> > point:
> >
> >
> > 1. Autoscaling
> > Supporting a (min, max) concurrency range is a great idea. However,
> > auto-scaling
> > involves much more than just the API surface — it requires discussion
> > around
> > scaling policies, coordination with the scheduler, metric-driven
> triggers,
> > and more.
> > To keep this FLIP focused, I'd like to defer that to a follow-up. Fixed
> > parallelism is
> > a common and necessary capability on its own, and the first version will
> > support
> > that. The interfaces can be extended later to include auto-scaling
> support
> > without
> > breaking changes.
> >
> >
> > 2. Async invocation & end-to-end examples
> > This is a good point. The `RosClient` already provides an async
> interface,
> > so using
> > it with `AsyncDataStream` is indeed more natural and idiomatic. I will
> add
> > an
> > asynchronous invocation example in the proposal. Regarding a complete
> > end-to-end
> > inference example: at the Java API level, it will still be primarily
> > conceptual, since
> > practical model inference examples really need Python API support (e.g.,
> > loading a
> > PyTorch or ONNX model). I will refine the existing example to make the
> > overall
> > invocation chain as close to a real-world scenario as possible.
> >
> >
> > 3. GPU as a resource
> > This is indeed a key concern. The `extendedResource` API in the original
> > proposal
> > comes from Flink's existing resource model (introduced by FLIP-108).
> > However, I
> > agree that GPU deserves first-class treatment rather than being accessed
> > through a
> > generic extension mechanism. Promoting GPU to a first-class resource will
> > involve
> > changes to both the API and the scheduling layer, which we plan to
> address
> > in a
> > dedicated FLIP. For now, I will remove the `extendedResource` API from
> > `RosDescriptor` to keep this FLIP focused on the RPC service primitive
> > itself. The
> > resource declaration API can be extended once the GPU-specific FLIP is in
> > place.
> >
> >
> > 4. Serial method invocation
> > Agreed. I will make the serial invocation semantics more prominent — both
> > in the
> > FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
> >
> >
> > 5. Why synchronous methods?
> > This is a key question. The initial design chose synchronous methods for
> > simplicity
> > — users don't need to worry about thread safety, and concurrency is
> > achieved by
> > scaling out instances (parallelism). However, I recognize that
> > asynchronous
> > processing is a critical requirement for certain scenarios (e.g., when
> the
> > RPC service
> > performs expensive operations that can be parallelized internally). I am
> > planning to
> > extend the current design to support asynchronous method execution: if a
> > user's
> > method returns a `CompletableFuture`, the framework will recognize it and
> > wait for
> > the future to complete before sending the response, without blocking the
> > invoke
> > thread. This allows the invoke thread to continue dispatching subsequent
> > requests,
> > while the user controls internal concurrency. Importantly,
> > `processRequest` calls
> > remain serial — the async behavior is purely opt-in.
> >
> >
> > 6. SQL UDF usage
> > The current design is primarily oriented toward the DataStream API, as it
> > provides
> > the foundational building blocks. I agree that SQL and Table API support
> > is an
> > important aspect to consider as well. On the invocation side, both Table
> > API and
> > SQL UDFs share the same runtime path through `FunctionContext`, so
> > enabling
> > RPC service access from UDFs is straightforward — I will add a
> > `getRosClient`
> > method to `FunctionContext` and include that change in this FLIP.
> > The real challenge for SQL lies on the definition side: how to define and
> > register
> > an RPC service using pure SQL syntax, which would require SQL grammar
> > extensions and planner changes. I think that part would be better
> > addressed in
> > a dedicated follow-up FLIP to give it the attention it deserves.
> >
> >
> > 7. Service sharing across jobs & lifecycle
> > In this FLIP, the RPC service is defined as part of the job (via
> > `env.addRpcService()`
> > before `env.execute()`). It cannot be accessed across jobs, and it is
> > released when
> > the job reaches a terminal state. I believe your question is more
> relevant
> > to the
> > "Shared RPC Service" mentioned in the Follow-up Tasks section. While it's
> > out of
> > scope for this FLIP, let me briefly describe the envisioned design: a
> > shared RPC
> > service would run as an independent job within the same Application,
> > accessible by
> > all jobs in that Application. It would terminate when the Application
> > ends, or could be
> > stopped manually via a user-facing API.
> > The embedded (per-job) service model discussed in this FLIP is simpler to
> > manage
> > and well-suited for many streaming scenarios, which is why we focus on it
> > first.
> > Importantly, both the embedded and shared modes will reuse the same set
> of
> > deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`, etc.)
> > — the
> > difference is only in lifecycle management and visibility scope.
> >
> >
> >
> >
> > I hope the above addresses your questions. I will update the proposal to
> > reflect the
> > changes mentioned above. Please feel free to follow up if you have any
> > further
> > questions or suggestions.
> >
> >
> > Best,
> > Yi
> >
> >
> >
> > At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
> > >HI,YI
> > >
> > >One more question I'd like to raise is around how the service behaves
> when
> > >it's referenced by more than one job. If two jobs are submitted within
> the
> > >same application and both reference the same RpcOperator service, is
> that
> > >service shared across them, and what is the expected behavior in that
> > case?
> > >I'm also curious how this plays out for bounded (batch) jobs in
> > particular:
> > >if one job completes while the other is still running, what does the
> > >service lifecycle look like — is the service torn down as soon as the
> > first
> > >job finishes, and if so, how does that affect the job that's still
> > running?
> > >
> > >Best,
> > >Guowei
> > >
> > >
> > >On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
> > >
> > >> Hi,YI
> > >>
> > >> Thanks for putting together this FLIP — it's a well-thought-out
> > proposal,
> > >> and the RpcOperator Service is a really valuable addition for
> > AI/inference
> > >> workloads on Flink. I have a few questions and suggestions:
> > >>
> > >>
> > >> 1. Autoscaling
> > >> How is scale-up/scale-down expressed in this design? Could you
> consider
> > >> allowing concurrency to be configured as a (min, max) range? Currently
> > it
> > >> appears only min = max (i.e. fixed parallelism) is supported.
> > >>
> > >>
> > >> 2. Async invocation & end-to-end examples
> > >> The example shows a synchronous invocation style. I'd suggest also
> > >> providing an asynchronous invocation example — or even just a scaffold
> > that
> > >> exposes only the RpcOperator's handle and method. In addition, it
> would
> > be
> > >> best to include a complete end-to-end example of invoking a model (CPU
> > >> inference would be fine).
> > >>
> > >>
> > >> 3. GPU as a resource
> > >> GPU is surely a common/first-class resource. Accessing it here through
> > an
> > >> extend mechanism — wouldn't that mislead users? It would seem more
> > >> natural to declare GPU as a standard resource rather than going
> through
> > an
> > >> extension.
> > >>
> > >>
> > >> 4. Serial method invocation
> > >> Each method of an RpcOperator is invoked serially. I'd suggest stating
> > >> this explicitly in both the documentation and the code comments, so
> that
> > >> users — or an AI — can be clearly aware of it.
> > >>
> > >>
> > >> 5. Why synchronous methods?
> > >> Why is every method in RpcOperator a synchronous call? Does this imply
> > >> that no model inference supports concurrent invocation?
> > >>
> > >>
> > >> 6. SQL UDF usage
> > >> How would SQL UDF users invoke this capability? I'd suggest providing
> a
> > >> complete example for this as well.
> > >>
> > >>
> > >> Thanks again for the great work!
> > >>
> > >> Best,
> > >> Guowei
> > >>
> > >>
> > >> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <dockerzhang@apache.org
> >
> > >> wrote:
> > >>
> > >>> Hi Yi, and Flink Community,
> > >>>
> > >>> Thanks for bringing up this excellent proposal. I am fully in favor
> of
> > >>> FLIP-582.
> > >>>
> > >>> In our production workloads, especially regarding large-scale AI
> > >>> inference,
> > >>> the tight coupling of the data plane and compute units has always
> been
> > a
> > >>> major pain point. If an inference subtask fails due to a GPU OOM or
> > driver
> > >>> issue, triggering a global rollback is incredibly expensive since
> model
> > >>> reloading takes minutes.
> > >>>
> > >>> The introduction of the RpcOperator Service as a first-class
> primitive
> > >>> masterfully decouples the heavy inference tasks from the mainstream
> > >>> topology. The fault isolation, independent scaling, and stateless
> > design
> > >>> perfectly match the requirements of modern AI-oriented data
> processing.
> > >>>
> > >>> This is a clean and robust architecture. Looking forward to seeing
> this
> > >>> merged!
> > >>>
> > >>>
> > >>> Best wishes,
> > >>> Charles Zhang
> > >>> from Apache InLong
> > >>>
> > >>>
> > >>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
> > >>>
> > >>> > Hi everyone,
> > >>> >
> > >>> >
> > >>> >
> > >>> > I would like to start a discussion on FLIP-582: Support RpcOperator
> > >>> > Service [1].
> > >>> >
> > >>> >
> > >>> > AI-oriented workloads like multimodal data processing and model
> > >>> inference
> > >>> > are
> > >>> > growing rapidly in recent years. These workloads are characterized
> by
> > >>> > expensive
> > >>> > resources (GPUs) and high initialization costs (seconds to minutes
> > for
> > >>> > model
> > >>> > loading). In today's Flink, embedding them in the data plane
> couples
> > >>> their
> > >>> > parallelism and failover with surrounding operators; deploying them
> > as
> > >>> > external
> > >>> > services disconnects their lifecycle from the job and doubles
> > >>> operational
> > >>> > overhead.
> > >>> >
> > >>> >
> > >>> > This FLIP introduces RpcOperator Service — a framework-level
> > primitive
> > >>> > that runs
> > >>> > user-defined compute as RPC services in an independent Pipelined
> > Region
> > >>> > within
> > >>> > the Flink job. Because the service is isolated at the scheduling
> > level,
> > >>> it
> > >>> > can achieve
> > >>> > fault isolation, independent scaling, and dedicated resource
> > allocation.
> > >>> > As a native
> > >>> > Flink primitive, it also lays the foundation for automatic flow
> > control,
> > >>> > flexible load
> > >>> > balancing, and coordinated auto-scaling — all without introducing
> > >>> external
> > >>> > infrastructure or additional operational burden.
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > Looking forward to your feedback and suggestions!
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > [1]
> > >>> >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > Best Regards,
> > >>> > Yi Zhang
> > >>>
> > >>
> >
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Gyula,
Thank you so much for your feedback and the really thoughtful questions. They’ve been
very valuable in helping me think more clearly about the scope and direction of this
proposal.
1. Guidance: RpcOperator vs StreamOperator
This is indeed a key question for users. The `StreamOperator` is a long-established API
in Flink, backed by a mature and efficient data transfer mechanism, state management,
and checkpoint infrastructure. RpcOperator is **not** intended to replace it — rather, it
serves as a complement for specific scenarios.
As described in the Motivation section, RpcOperator targets workloads that are
**computation-heavy, expensive to initialize, and require costly resources**. In such
scenarios, fault isolation and flexible scaling are the core requirements, which is precisely
the design rationale and key advantage of RpcOperator. We recommend users adopt
RpcOperator for these types of workloads — typical examples include multimodal data
processing, model inference, and similar use cases. Additionally, given the design and
implementation complexity, RpcOperator does not support stateful computation at this
stage.
On the other hand, if the processing logic is lightweight, initialization is fast, resources
are inexpensive, or stateful processing is required, then staying with `StreamOperator`
is likely the better choice — it avoids the RPC transport overhead that RpcOperator
introduces.
2. Performance comparison
I am currently running a set of benchmarks. Preliminary findings show that the
performance impact of RpcOperator is influenced by two key factors:
**object size** and **per-request processing time**.
When the object size is small (e.g., ~10KB) and the per-request processing time is
relatively long (e.g., >5ms), the overhead introduced by RpcOperator is nearly negligible
— less than 2%.
For short-latency processing (which is typically not the target scenario for RpcOperator)
or large object transfers, the overhead becomes non-negligible. The large-object case
is partly related to our current choice of Pekko as the underlying transport layer;
replacing it with a more efficient transport for large payloads is a possible optimization
direction.
Again, the isolation advantages of RpcOperator are not free. Our goal is to deliver this
capability precisely where it matters most — while keeping performance impact within
acceptable bounds.
3. Chaining RpcOperators & RpcOperator-only jobs
Chaining RpcOperator calls — where one RpcOperator invokes another — should be
feasible. This can be achieved by exposing `RosClient` through the `Context` available
to each RpcOperator. However, users should be aware that this may introduce
cascading RPC overhead.
Regarding the idea of a fully RpcOperator-based job: this relates to the
**cross-job shared service** concept mentioned in the Follow-up Tasks section — where
a dedicated job provides only an RPC service with no data-plane operators, and other
jobs share that service during their data processing. I believe this is a promising
extension. I am now planning to include the interfaces for this capability in this FLIP and
will update the proposal to reflect these changes.
Thank you again for the valuable feedback and inspiration! Please feel free to follow up
if you have any further questions or suggestions.
Best,
Yi
At 2026-06-09 13:18:39, "Gyula Fóra" <gy...@gmail.com> wrote:
>Hey!
>This indeed looks very promising and powerful. At the same time it also
>introduces a major new way of defining processing logic ("doing the same
>thing").
>To this end I have some high level questions that I couldn't answer based
>on the proposal and discussion related to how such a giant new piece fits
>into the existing API and ways of working for our users.
>
>1. How will users know when to use the RPC operator vs the
>existing datastream APIs, what will be our guidance?
>2. How does the performance (throughput, latency) compare realistically to
>a regular datastream operator? (I guess this relates to question 1 in a way)
>3. Can RPC operators call other RPC operators? If I have a stateless job,
>why not turn all my operators into RPC operators to get the independent
>scaling/recovery benefits.
>
>Thank you!
>Gyula
>
>On Fri, Jun 5, 2026 at 4:27 AM Yi Zhang <zh...@163.com> wrote:
>
>> Hi Gen,
>>
>>
>> Thanks for the thorough review and the insightful implementation-level
>> questions!
>> Here are my responses to each point:
>>
>>
>> 1. Heterogeneous resources
>> I fully agree that heterogeneous resource scheduling is a critical part of
>> realizing
>> the full value of decoupled GPU/CPU scaling. However, as mentioned in my
>> reply
>> to Hongshun's Q4, this is fundamentally a resource allocation and
>> scheduling
>> problem that goes beyond the scope of this FLIP. We plan to address it in
>> a
>> dedicated GPU and resource scheduling FLIP. The current design already
>> uses
>> separate SlotSharingGroups for ROS services, and the interface is designed
>> to
>> be extended without breaking changes when the resource scheduling FLIP
>> lands.
>>
>>
>> 2.1 RpcTask Design
>> This is a very thoughtful architectural suggestion, and I appreciate the
>> detailed
>> reasoning from your StreamTask experience. You are right that the current
>> FLIP
>> focuses more on the interfaces, and some implementation details like
>> Mailbox
>> integration deserve further discussion.
>> The key reason for not introducing Mailbox in the current RpcTask is that
>> the
>> RpcOperator does not yet have a strong enough need for it — especially
>> given
>> the stateless design, which already covers a wide range of scenarios. The
>> simple queue + single-threaded run loop is sufficient to implement all
>> currently
>> required capabilities. This keeps the implementation minimal and easy to
>> reason about.
>> That said, as the RpcOperator's capabilities expand in the future — such
>> as
>> supporting checkpoint participation — introducing Mailbox would be a
>> natural
>> and valuable evolution. Importantly, this is a purely internal refactoring
>> that does
>> not affect any user-facing interfaces, so it can be adopted incrementally
>> when
>> the need arises.
>>
>>
>> 2.2. Communication implementation
>> This concern is similar to Xintong's question on the communication layer.
>> The
>> first version directly reuses Flink's existing Pekko RPC infrastructure,
>> but in the
>> longer term, introducing an alternative transport layer such as gRPC —
>> which
>> natively supports streaming — would address both the large-payload concern
>> and the unfold/flatMap scenario you raised. This can be considered as part
>> of
>> the communication layer evolution, and the upper-level `RosClient` /
>> `RpcOperator` interfaces are designed to be transport-agnostic.
>>
>>
>> 2.3. Client-side implementation
>> This is indeed a key question. Let me clarify how the client-side
>> checkpoint
>> interaction works. The `RosClient` provides an asynchronous interface
>> (`CompletableFuture`), and users have two usage patterns:
>>
>>
>> a. Synchronous usage (e.g., `client.request(...).get()` in a
>> `ProcessFunction`):
>> The request completes before `processElement` returns, so there are no
>> in-flight requests at checkpoint time. Checkpoint is not a concern in this
>> case.
>>
>>
>> b. Asynchronous usage: Note that users should use Flink's
>> `AsyncDataStream`
>> with `RichAsyncFunction`, which already has built-in handling for pending
>> requests during checkpoints. The `RosClient` itself does not need to
>> handle
>> any pending request recovery, just as any other async HTTP or RPC client
>> used within `AsyncFunction` does not.
>>
>>
>> In other words, the checkpoint semantics are not a new problem introduced
>> by ROS — they are handled by the existing `AsyncDataStream` framework,
>> which users are already expected to use for asynchronous invocation
>> patterns.
>> I will update the FLIP to emphasize this point more clearly.
>>
>>
>>
>>
>> 3. ROS as container, not service
>> This is a great direction to explore. In fact, the current `RpcOperator`
>> interface
>> is flexible enough to support this container-style usage: a user could
>> implement
>> `open()` to start an external service process and the `processRequest()`
>> method
>> could simply proxy to the external service, or — as you suggest — be
>> bypassed
>> entirely if the data plane communicates directly with the external service
>> endpoint. This is a viable and valuable pattern. One trade-off to note is
>> that
>> some of ROS's built-in capabilities — such as framework-level load
>> balancing
>> — would not be utilized in this mode, since traffic bypasses the RPC
>> request
>> queue. Users would need to evaluate whether the external service provides
>> equivalent capabilities or whether the direct-access performance benefit
>> outweighs the loss of framework-level features.
>> Thank you for raising this — it validates an important use case that the
>> current
>> architecture can accommodate.
>>
>>
>> 4. Naming
>> There is indeed a naming inconsistency — `addRpcService` can be renamed
>> to `addRpcOperatorService` to align with the other interfaces. Thanks for
>> pointing this out.
>>
>>
>> I hope these responses address your concerns. Please feel free to follow
>> up
>> with any further questions or suggestions.
>>
>>
>>
>>
>> Best,
>> Yi
>>
>>
>>
>> At 2026-06-02 11:15:24, "Gen Luo" <lu...@gmail.com> wrote:
>> >Hi Yi,
>> >
>> >Thanks for this impressive proposal. Adding an RPC service deployment
>> >primitive can greatly expand Flink's capability scope and flexibility,
>> with
>> >clear benefits for AI workloads and agent applications. It can also play a
>> >role in scenarios like state management or data lake compaction — relevant
>> >to but decoupled from the data pipeline. This is a very promising
>> >direction. That said, I still have some questions.
>> >
>> >1. My biggest concern is heterogeneous resources, echoing Hongshun's Q4.
>> >Flink today uses homogeneous TM resources sliced for different task
>> >requirements, but decoupling GPU/CPU resources and enabling flexible
>> >independent scaling is one of the core values of this FLIP — and the
>> >current resource allocation approach cannot support this. We likely need
>> >new implementations for the resource allocation component.
>> >
>> >2. Some components are described only at the idea level without
>> >implementation details. Hopefully these can be supplemented later or
>> >discussed in dedicated threads. A few worth discussing:
>> >
>> > - RpcTask: It's currently described as a simple single-threaded loop,
>> >with no plan to use Mailbox or OperatorChain. But if we want to support
>> >async, checkpoint, etc. later, introducing Mailbox would be valuable —
>> >handle the input queue as defaultAction, with async/CP/system messages as
>> >mails. This keeps semantics consistent with StreamTask and reduces
>> >maintenance burden. System events like FO-triggered request cancellation
>> >can also be prioritized via mailbox. OperatorChain is what
>> >SubtaskCheckpointCoordinator directly contacts, so having RpcOperator as
>> >the only node in the chain wouldn't add much complexity and lowers the
>> cost
>> >when CP support comes.
>> >
>> > - Communication implementation: Will it reuse the Pekko + proxy
>> >mechanism? Pekko has risks with large message bodies (OOM, silently drop
>> >oversized frame, etc.). Since ROS is user-facing, we can't fully avoid
>> >this, and large-model scenarios often carry sizable context — needs
>> careful
>> >handling. Also, the current design doesn't include table-function-style
>> >interfaces for unfolding data, so a large unfolded result has to be
>> >returned as a single response, inflating per-request payload size and
>> >amplifying the same risk.
>> >
>> > - Client-side implementation: Only the interface is described, no
>> >implementation details. Checkpoint is the prominent issue — since ROS is
>> >stateless by design, state recording falls to the client. Do we go with
>> >drain or record? Drain may take a long time; record on a keyed stream may
>> >cause request distribution confusion after parallelism changes.
>> >
>> >3. ROS as container, not service: Is this allowed? If the system itself
>> has
>> >service capabilities — especially non-Java stacks (agents, LLMs, also
>> >Python remote functions) — forwarding requests through the Flink Java
>> layer
>> >adds unnecessary overhead. In that case RpcOperator would only handle
>> >lifecycle, while the framework manages resource scheduling and service
>> >discovery without carrying business traffic.
>> >
>> >nit: addRpcService is inconsistent with other places that use RpcOperator
>> /
>> >the ros keyword. Worth unifying.
>> >
>> >Best,
>> >Gen
>> >
>> >On Mon, Jun 1, 2026 at 2:02 PM Yi Zhang <zh...@163.com> wrote:
>> >
>> >> Hi Guowei,
>> >>
>> >>
>> >> Thanks for the thoughtful review and valuable feedback! Your questions
>> are
>> >> insightful
>> >> and have directly driven refinements to the proposal. Here are my
>> >> responses to each
>> >> point:
>> >>
>> >>
>> >> 1. Autoscaling
>> >> Supporting a (min, max) concurrency range is a great idea. However,
>> >> auto-scaling
>> >> involves much more than just the API surface — it requires discussion
>> >> around
>> >> scaling policies, coordination with the scheduler, metric-driven
>> triggers,
>> >> and more.
>> >> To keep this FLIP focused, I'd like to defer that to a follow-up. Fixed
>> >> parallelism is
>> >> a common and necessary capability on its own, and the first version will
>> >> support
>> >> that. The interfaces can be extended later to include auto-scaling
>> support
>> >> without
>> >> breaking changes.
>> >>
>> >>
>> >> 2. Async invocation & end-to-end examples
>> >> This is a good point. The `RosClient` already provides an async
>> interface,
>> >> so using
>> >> it with `AsyncDataStream` is indeed more natural and idiomatic. I will
>> add
>> >> an
>> >> asynchronous invocation example in the proposal. Regarding a complete
>> >> end-to-end
>> >> inference example: at the Java API level, it will still be primarily
>> >> conceptual, since
>> >> practical model inference examples really need Python API support (e.g.,
>> >> loading a
>> >> PyTorch or ONNX model). I will refine the existing example to make the
>> >> overall
>> >> invocation chain as close to a real-world scenario as possible.
>> >>
>> >>
>> >> 3. GPU as a resource
>> >> This is indeed a key concern. The `extendedResource` API in the original
>> >> proposal
>> >> comes from Flink's existing resource model (introduced by FLIP-108).
>> >> However, I
>> >> agree that GPU deserves first-class treatment rather than being accessed
>> >> through a
>> >> generic extension mechanism. Promoting GPU to a first-class resource
>> will
>> >> involve
>> >> changes to both the API and the scheduling layer, which we plan to
>> address
>> >> in a
>> >> dedicated FLIP. For now, I will remove the `extendedResource` API from
>> >> `RosDescriptor` to keep this FLIP focused on the RPC service primitive
>> >> itself. The
>> >> resource declaration API can be extended once the GPU-specific FLIP is
>> in
>> >> place.
>> >>
>> >>
>> >> 4. Serial method invocation
>> >> Agreed. I will make the serial invocation semantics more prominent —
>> both
>> >> in the
>> >> FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
>> >>
>> >>
>> >> 5. Why synchronous methods?
>> >> This is a key question. The initial design chose synchronous methods for
>> >> simplicity
>> >> — users don't need to worry about thread safety, and concurrency is
>> >> achieved by
>> >> scaling out instances (parallelism). However, I recognize that
>> >> asynchronous
>> >> processing is a critical requirement for certain scenarios (e.g., when
>> the
>> >> RPC service
>> >> performs expensive operations that can be parallelized internally). I am
>> >> planning to
>> >> extend the current design to support asynchronous method execution: if a
>> >> user's
>> >> method returns a `CompletableFuture`, the framework will recognize it
>> and
>> >> wait for
>> >> the future to complete before sending the response, without blocking the
>> >> invoke
>> >> thread. This allows the invoke thread to continue dispatching subsequent
>> >> requests,
>> >> while the user controls internal concurrency. Importantly,
>> >> `processRequest` calls
>> >> remain serial — the async behavior is purely opt-in.
>> >>
>> >>
>> >> 6. SQL UDF usage
>> >> The current design is primarily oriented toward the DataStream API, as
>> it
>> >> provides
>> >> the foundational building blocks. I agree that SQL and Table API support
>> >> is an
>> >> important aspect to consider as well. On the invocation side, both Table
>> >> API and
>> >> SQL UDFs share the same runtime path through `FunctionContext`, so
>> >> enabling
>> >> RPC service access from UDFs is straightforward — I will add a
>> >> `getRosClient`
>> >> method to `FunctionContext` and include that change in this FLIP.
>> >> The real challenge for SQL lies on the definition side: how to define
>> and
>> >> register
>> >> an RPC service using pure SQL syntax, which would require SQL grammar
>> >> extensions and planner changes. I think that part would be better
>> >> addressed in
>> >> a dedicated follow-up FLIP to give it the attention it deserves.
>> >>
>> >>
>> >> 7. Service sharing across jobs & lifecycle
>> >> In this FLIP, the RPC service is defined as part of the job (via
>> >> `env.addRpcService()`
>> >> before `env.execute()`). It cannot be accessed across jobs, and it is
>> >> released when
>> >> the job reaches a terminal state. I believe your question is more
>> relevant
>> >> to the
>> >> "Shared RPC Service" mentioned in the Follow-up Tasks section. While
>> it's
>> >> out of
>> >> scope for this FLIP, let me briefly describe the envisioned design: a
>> >> shared RPC
>> >> service would run as an independent job within the same Application,
>> >> accessible by
>> >> all jobs in that Application. It would terminate when the Application
>> >> ends, or could be
>> >> stopped manually via a user-facing API.
>> >> The embedded (per-job) service model discussed in this FLIP is simpler
>> to
>> >> manage
>> >> and well-suited for many streaming scenarios, which is why we focus on
>> it
>> >> first.
>> >> Importantly, both the embedded and shared modes will reuse the same set
>> of
>> >> deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`,
>> etc.)
>> >> — the
>> >> difference is only in lifecycle management and visibility scope.
>> >>
>> >>
>> >>
>> >>
>> >> I hope the above addresses your questions. I will update the proposal to
>> >> reflect the
>> >> changes mentioned above. Please feel free to follow up if you have any
>> >> further
>> >> questions or suggestions.
>> >>
>> >>
>> >> Best,
>> >> Yi
>> >>
>> >>
>> >>
>> >> At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
>> >> >HI,YI
>> >> >
>> >> >One more question I'd like to raise is around how the service behaves
>> when
>> >> >it's referenced by more than one job. If two jobs are submitted within
>> the
>> >> >same application and both reference the same RpcOperator service, is
>> that
>> >> >service shared across them, and what is the expected behavior in that
>> >> case?
>> >> >I'm also curious how this plays out for bounded (batch) jobs in
>> >> particular:
>> >> >if one job completes while the other is still running, what does the
>> >> >service lifecycle look like — is the service torn down as soon as the
>> >> first
>> >> >job finishes, and if so, how does that affect the job that's still
>> >> running?
>> >> >
>> >> >Best,
>> >> >Guowei
>> >> >
>> >> >
>> >> >On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com>
>> wrote:
>> >> >
>> >> >> Hi,YI
>> >> >>
>> >> >> Thanks for putting together this FLIP — it's a well-thought-out
>> >> proposal,
>> >> >> and the RpcOperator Service is a really valuable addition for
>> >> AI/inference
>> >> >> workloads on Flink. I have a few questions and suggestions:
>> >> >>
>> >> >>
>> >> >> 1. Autoscaling
>> >> >> How is scale-up/scale-down expressed in this design? Could you
>> consider
>> >> >> allowing concurrency to be configured as a (min, max) range?
>> Currently
>> >> it
>> >> >> appears only min = max (i.e. fixed parallelism) is supported.
>> >> >>
>> >> >>
>> >> >> 2. Async invocation & end-to-end examples
>> >> >> The example shows a synchronous invocation style. I'd suggest also
>> >> >> providing an asynchronous invocation example — or even just a
>> scaffold
>> >> that
>> >> >> exposes only the RpcOperator's handle and method. In addition, it
>> would
>> >> be
>> >> >> best to include a complete end-to-end example of invoking a model
>> (CPU
>> >> >> inference would be fine).
>> >> >>
>> >> >>
>> >> >> 3. GPU as a resource
>> >> >> GPU is surely a common/first-class resource. Accessing it here
>> through
>> >> an
>> >> >> extend mechanism — wouldn't that mislead users? It would seem more
>> >> >> natural to declare GPU as a standard resource rather than going
>> through
>> >> an
>> >> >> extension.
>> >> >>
>> >> >>
>> >> >> 4. Serial method invocation
>> >> >> Each method of an RpcOperator is invoked serially. I'd suggest
>> stating
>> >> >> this explicitly in both the documentation and the code comments, so
>> that
>> >> >> users — or an AI — can be clearly aware of it.
>> >> >>
>> >> >>
>> >> >> 5. Why synchronous methods?
>> >> >> Why is every method in RpcOperator a synchronous call? Does this
>> imply
>> >> >> that no model inference supports concurrent invocation?
>> >> >>
>> >> >>
>> >> >> 6. SQL UDF usage
>> >> >> How would SQL UDF users invoke this capability? I'd suggest
>> providing a
>> >> >> complete example for this as well.
>> >> >>
>> >> >>
>> >> >> Thanks again for the great work!
>> >> >>
>> >> >> Best,
>> >> >> Guowei
>> >> >>
>> >> >>
>> >> >> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <
>> dockerzhang@apache.org>
>> >> >> wrote:
>> >> >>
>> >> >>> Hi Yi, and Flink Community,
>> >> >>>
>> >> >>> Thanks for bringing up this excellent proposal. I am fully in favor
>> of
>> >> >>> FLIP-582.
>> >> >>>
>> >> >>> In our production workloads, especially regarding large-scale AI
>> >> >>> inference,
>> >> >>> the tight coupling of the data plane and compute units has always
>> been
>> >> a
>> >> >>> major pain point. If an inference subtask fails due to a GPU OOM or
>> >> driver
>> >> >>> issue, triggering a global rollback is incredibly expensive since
>> model
>> >> >>> reloading takes minutes.
>> >> >>>
>> >> >>> The introduction of the RpcOperator Service as a first-class
>> primitive
>> >> >>> masterfully decouples the heavy inference tasks from the mainstream
>> >> >>> topology. The fault isolation, independent scaling, and stateless
>> >> design
>> >> >>> perfectly match the requirements of modern AI-oriented data
>> processing.
>> >> >>>
>> >> >>> This is a clean and robust architecture. Looking forward to seeing
>> this
>> >> >>> merged!
>> >> >>>
>> >> >>>
>> >> >>> Best wishes,
>> >> >>> Charles Zhang
>> >> >>> from Apache InLong
>> >> >>>
>> >> >>>
>> >> >>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>> >> >>>
>> >> >>> > Hi everyone,
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > I would like to start a discussion on FLIP-582: Support
>> RpcOperator
>> >> >>> > Service [1].
>> >> >>> >
>> >> >>> >
>> >> >>> > AI-oriented workloads like multimodal data processing and model
>> >> >>> inference
>> >> >>> > are
>> >> >>> > growing rapidly in recent years. These workloads are
>> characterized by
>> >> >>> > expensive
>> >> >>> > resources (GPUs) and high initialization costs (seconds to minutes
>> >> for
>> >> >>> > model
>> >> >>> > loading). In today's Flink, embedding them in the data plane
>> couples
>> >> >>> their
>> >> >>> > parallelism and failover with surrounding operators; deploying
>> them
>> >> as
>> >> >>> > external
>> >> >>> > services disconnects their lifecycle from the job and doubles
>> >> >>> operational
>> >> >>> > overhead.
>> >> >>> >
>> >> >>> >
>> >> >>> > This FLIP introduces RpcOperator Service — a framework-level
>> >> primitive
>> >> >>> > that runs
>> >> >>> > user-defined compute as RPC services in an independent Pipelined
>> >> Region
>> >> >>> > within
>> >> >>> > the Flink job. Because the service is isolated at the scheduling
>> >> level,
>> >> >>> it
>> >> >>> > can achieve
>> >> >>> > fault isolation, independent scaling, and dedicated resource
>> >> allocation.
>> >> >>> > As a native
>> >> >>> > Flink primitive, it also lays the foundation for automatic flow
>> >> control,
>> >> >>> > flexible load
>> >> >>> > balancing, and coordinated auto-scaling — all without introducing
>> >> >>> external
>> >> >>> > infrastructure or additional operational burden.
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > Looking forward to your feedback and suggestions!
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > [1]
>> >> >>> >
>> >> >>> >
>> >> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > Best Regards,
>> >> >>> > Yi Zhang
>> >> >>>
>> >> >>
>> >>
>>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Gyula Fóra <gy...@gmail.com>.
Hey!
This indeed looks very promising and powerful. At the same time it also
introduces a major new way of defining processing logic ("doing the same
thing").
To this end I have some high level questions that I couldn't answer based
on the proposal and discussion related to how such a giant new piece fits
into the existing API and ways of working for our users.
1. How will users know when to use the RPC operator vs the
existing datastream APIs, what will be our guidance?
2. How does the performance (throughput, latency) compare realistically to
a regular datastream operator? (I guess this relates to question 1 in a way)
3. Can RPC operators call other RPC operators? If I have a stateless job,
why not turn all my operators into RPC operators to get the independent
scaling/recovery benefits.
Thank you!
Gyula
On Fri, Jun 5, 2026 at 4:27 AM Yi Zhang <zh...@163.com> wrote:
> Hi Gen,
>
>
> Thanks for the thorough review and the insightful implementation-level
> questions!
> Here are my responses to each point:
>
>
> 1. Heterogeneous resources
> I fully agree that heterogeneous resource scheduling is a critical part of
> realizing
> the full value of decoupled GPU/CPU scaling. However, as mentioned in my
> reply
> to Hongshun's Q4, this is fundamentally a resource allocation and
> scheduling
> problem that goes beyond the scope of this FLIP. We plan to address it in
> a
> dedicated GPU and resource scheduling FLIP. The current design already
> uses
> separate SlotSharingGroups for ROS services, and the interface is designed
> to
> be extended without breaking changes when the resource scheduling FLIP
> lands.
>
>
> 2.1 RpcTask Design
> This is a very thoughtful architectural suggestion, and I appreciate the
> detailed
> reasoning from your StreamTask experience. You are right that the current
> FLIP
> focuses more on the interfaces, and some implementation details like
> Mailbox
> integration deserve further discussion.
> The key reason for not introducing Mailbox in the current RpcTask is that
> the
> RpcOperator does not yet have a strong enough need for it — especially
> given
> the stateless design, which already covers a wide range of scenarios. The
> simple queue + single-threaded run loop is sufficient to implement all
> currently
> required capabilities. This keeps the implementation minimal and easy to
> reason about.
> That said, as the RpcOperator's capabilities expand in the future — such
> as
> supporting checkpoint participation — introducing Mailbox would be a
> natural
> and valuable evolution. Importantly, this is a purely internal refactoring
> that does
> not affect any user-facing interfaces, so it can be adopted incrementally
> when
> the need arises.
>
>
> 2.2. Communication implementation
> This concern is similar to Xintong's question on the communication layer.
> The
> first version directly reuses Flink's existing Pekko RPC infrastructure,
> but in the
> longer term, introducing an alternative transport layer such as gRPC —
> which
> natively supports streaming — would address both the large-payload concern
> and the unfold/flatMap scenario you raised. This can be considered as part
> of
> the communication layer evolution, and the upper-level `RosClient` /
> `RpcOperator` interfaces are designed to be transport-agnostic.
>
>
> 2.3. Client-side implementation
> This is indeed a key question. Let me clarify how the client-side
> checkpoint
> interaction works. The `RosClient` provides an asynchronous interface
> (`CompletableFuture`), and users have two usage patterns:
>
>
> a. Synchronous usage (e.g., `client.request(...).get()` in a
> `ProcessFunction`):
> The request completes before `processElement` returns, so there are no
> in-flight requests at checkpoint time. Checkpoint is not a concern in this
> case.
>
>
> b. Asynchronous usage: Note that users should use Flink's
> `AsyncDataStream`
> with `RichAsyncFunction`, which already has built-in handling for pending
> requests during checkpoints. The `RosClient` itself does not need to
> handle
> any pending request recovery, just as any other async HTTP or RPC client
> used within `AsyncFunction` does not.
>
>
> In other words, the checkpoint semantics are not a new problem introduced
> by ROS — they are handled by the existing `AsyncDataStream` framework,
> which users are already expected to use for asynchronous invocation
> patterns.
> I will update the FLIP to emphasize this point more clearly.
>
>
>
>
> 3. ROS as container, not service
> This is a great direction to explore. In fact, the current `RpcOperator`
> interface
> is flexible enough to support this container-style usage: a user could
> implement
> `open()` to start an external service process and the `processRequest()`
> method
> could simply proxy to the external service, or — as you suggest — be
> bypassed
> entirely if the data plane communicates directly with the external service
> endpoint. This is a viable and valuable pattern. One trade-off to note is
> that
> some of ROS's built-in capabilities — such as framework-level load
> balancing
> — would not be utilized in this mode, since traffic bypasses the RPC
> request
> queue. Users would need to evaluate whether the external service provides
> equivalent capabilities or whether the direct-access performance benefit
> outweighs the loss of framework-level features.
> Thank you for raising this — it validates an important use case that the
> current
> architecture can accommodate.
>
>
> 4. Naming
> There is indeed a naming inconsistency — `addRpcService` can be renamed
> to `addRpcOperatorService` to align with the other interfaces. Thanks for
> pointing this out.
>
>
> I hope these responses address your concerns. Please feel free to follow
> up
> with any further questions or suggestions.
>
>
>
>
> Best,
> Yi
>
>
>
> At 2026-06-02 11:15:24, "Gen Luo" <lu...@gmail.com> wrote:
> >Hi Yi,
> >
> >Thanks for this impressive proposal. Adding an RPC service deployment
> >primitive can greatly expand Flink's capability scope and flexibility,
> with
> >clear benefits for AI workloads and agent applications. It can also play a
> >role in scenarios like state management or data lake compaction — relevant
> >to but decoupled from the data pipeline. This is a very promising
> >direction. That said, I still have some questions.
> >
> >1. My biggest concern is heterogeneous resources, echoing Hongshun's Q4.
> >Flink today uses homogeneous TM resources sliced for different task
> >requirements, but decoupling GPU/CPU resources and enabling flexible
> >independent scaling is one of the core values of this FLIP — and the
> >current resource allocation approach cannot support this. We likely need
> >new implementations for the resource allocation component.
> >
> >2. Some components are described only at the idea level without
> >implementation details. Hopefully these can be supplemented later or
> >discussed in dedicated threads. A few worth discussing:
> >
> > - RpcTask: It's currently described as a simple single-threaded loop,
> >with no plan to use Mailbox or OperatorChain. But if we want to support
> >async, checkpoint, etc. later, introducing Mailbox would be valuable —
> >handle the input queue as defaultAction, with async/CP/system messages as
> >mails. This keeps semantics consistent with StreamTask and reduces
> >maintenance burden. System events like FO-triggered request cancellation
> >can also be prioritized via mailbox. OperatorChain is what
> >SubtaskCheckpointCoordinator directly contacts, so having RpcOperator as
> >the only node in the chain wouldn't add much complexity and lowers the
> cost
> >when CP support comes.
> >
> > - Communication implementation: Will it reuse the Pekko + proxy
> >mechanism? Pekko has risks with large message bodies (OOM, silently drop
> >oversized frame, etc.). Since ROS is user-facing, we can't fully avoid
> >this, and large-model scenarios often carry sizable context — needs
> careful
> >handling. Also, the current design doesn't include table-function-style
> >interfaces for unfolding data, so a large unfolded result has to be
> >returned as a single response, inflating per-request payload size and
> >amplifying the same risk.
> >
> > - Client-side implementation: Only the interface is described, no
> >implementation details. Checkpoint is the prominent issue — since ROS is
> >stateless by design, state recording falls to the client. Do we go with
> >drain or record? Drain may take a long time; record on a keyed stream may
> >cause request distribution confusion after parallelism changes.
> >
> >3. ROS as container, not service: Is this allowed? If the system itself
> has
> >service capabilities — especially non-Java stacks (agents, LLMs, also
> >Python remote functions) — forwarding requests through the Flink Java
> layer
> >adds unnecessary overhead. In that case RpcOperator would only handle
> >lifecycle, while the framework manages resource scheduling and service
> >discovery without carrying business traffic.
> >
> >nit: addRpcService is inconsistent with other places that use RpcOperator
> /
> >the ros keyword. Worth unifying.
> >
> >Best,
> >Gen
> >
> >On Mon, Jun 1, 2026 at 2:02 PM Yi Zhang <zh...@163.com> wrote:
> >
> >> Hi Guowei,
> >>
> >>
> >> Thanks for the thoughtful review and valuable feedback! Your questions
> are
> >> insightful
> >> and have directly driven refinements to the proposal. Here are my
> >> responses to each
> >> point:
> >>
> >>
> >> 1. Autoscaling
> >> Supporting a (min, max) concurrency range is a great idea. However,
> >> auto-scaling
> >> involves much more than just the API surface — it requires discussion
> >> around
> >> scaling policies, coordination with the scheduler, metric-driven
> triggers,
> >> and more.
> >> To keep this FLIP focused, I'd like to defer that to a follow-up. Fixed
> >> parallelism is
> >> a common and necessary capability on its own, and the first version will
> >> support
> >> that. The interfaces can be extended later to include auto-scaling
> support
> >> without
> >> breaking changes.
> >>
> >>
> >> 2. Async invocation & end-to-end examples
> >> This is a good point. The `RosClient` already provides an async
> interface,
> >> so using
> >> it with `AsyncDataStream` is indeed more natural and idiomatic. I will
> add
> >> an
> >> asynchronous invocation example in the proposal. Regarding a complete
> >> end-to-end
> >> inference example: at the Java API level, it will still be primarily
> >> conceptual, since
> >> practical model inference examples really need Python API support (e.g.,
> >> loading a
> >> PyTorch or ONNX model). I will refine the existing example to make the
> >> overall
> >> invocation chain as close to a real-world scenario as possible.
> >>
> >>
> >> 3. GPU as a resource
> >> This is indeed a key concern. The `extendedResource` API in the original
> >> proposal
> >> comes from Flink's existing resource model (introduced by FLIP-108).
> >> However, I
> >> agree that GPU deserves first-class treatment rather than being accessed
> >> through a
> >> generic extension mechanism. Promoting GPU to a first-class resource
> will
> >> involve
> >> changes to both the API and the scheduling layer, which we plan to
> address
> >> in a
> >> dedicated FLIP. For now, I will remove the `extendedResource` API from
> >> `RosDescriptor` to keep this FLIP focused on the RPC service primitive
> >> itself. The
> >> resource declaration API can be extended once the GPU-specific FLIP is
> in
> >> place.
> >>
> >>
> >> 4. Serial method invocation
> >> Agreed. I will make the serial invocation semantics more prominent —
> both
> >> in the
> >> FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
> >>
> >>
> >> 5. Why synchronous methods?
> >> This is a key question. The initial design chose synchronous methods for
> >> simplicity
> >> — users don't need to worry about thread safety, and concurrency is
> >> achieved by
> >> scaling out instances (parallelism). However, I recognize that
> >> asynchronous
> >> processing is a critical requirement for certain scenarios (e.g., when
> the
> >> RPC service
> >> performs expensive operations that can be parallelized internally). I am
> >> planning to
> >> extend the current design to support asynchronous method execution: if a
> >> user's
> >> method returns a `CompletableFuture`, the framework will recognize it
> and
> >> wait for
> >> the future to complete before sending the response, without blocking the
> >> invoke
> >> thread. This allows the invoke thread to continue dispatching subsequent
> >> requests,
> >> while the user controls internal concurrency. Importantly,
> >> `processRequest` calls
> >> remain serial — the async behavior is purely opt-in.
> >>
> >>
> >> 6. SQL UDF usage
> >> The current design is primarily oriented toward the DataStream API, as
> it
> >> provides
> >> the foundational building blocks. I agree that SQL and Table API support
> >> is an
> >> important aspect to consider as well. On the invocation side, both Table
> >> API and
> >> SQL UDFs share the same runtime path through `FunctionContext`, so
> >> enabling
> >> RPC service access from UDFs is straightforward — I will add a
> >> `getRosClient`
> >> method to `FunctionContext` and include that change in this FLIP.
> >> The real challenge for SQL lies on the definition side: how to define
> and
> >> register
> >> an RPC service using pure SQL syntax, which would require SQL grammar
> >> extensions and planner changes. I think that part would be better
> >> addressed in
> >> a dedicated follow-up FLIP to give it the attention it deserves.
> >>
> >>
> >> 7. Service sharing across jobs & lifecycle
> >> In this FLIP, the RPC service is defined as part of the job (via
> >> `env.addRpcService()`
> >> before `env.execute()`). It cannot be accessed across jobs, and it is
> >> released when
> >> the job reaches a terminal state. I believe your question is more
> relevant
> >> to the
> >> "Shared RPC Service" mentioned in the Follow-up Tasks section. While
> it's
> >> out of
> >> scope for this FLIP, let me briefly describe the envisioned design: a
> >> shared RPC
> >> service would run as an independent job within the same Application,
> >> accessible by
> >> all jobs in that Application. It would terminate when the Application
> >> ends, or could be
> >> stopped manually via a user-facing API.
> >> The embedded (per-job) service model discussed in this FLIP is simpler
> to
> >> manage
> >> and well-suited for many streaming scenarios, which is why we focus on
> it
> >> first.
> >> Importantly, both the embedded and shared modes will reuse the same set
> of
> >> deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`,
> etc.)
> >> — the
> >> difference is only in lifecycle management and visibility scope.
> >>
> >>
> >>
> >>
> >> I hope the above addresses your questions. I will update the proposal to
> >> reflect the
> >> changes mentioned above. Please feel free to follow up if you have any
> >> further
> >> questions or suggestions.
> >>
> >>
> >> Best,
> >> Yi
> >>
> >>
> >>
> >> At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
> >> >HI,YI
> >> >
> >> >One more question I'd like to raise is around how the service behaves
> when
> >> >it's referenced by more than one job. If two jobs are submitted within
> the
> >> >same application and both reference the same RpcOperator service, is
> that
> >> >service shared across them, and what is the expected behavior in that
> >> case?
> >> >I'm also curious how this plays out for bounded (batch) jobs in
> >> particular:
> >> >if one job completes while the other is still running, what does the
> >> >service lifecycle look like — is the service torn down as soon as the
> >> first
> >> >job finishes, and if so, how does that affect the job that's still
> >> running?
> >> >
> >> >Best,
> >> >Guowei
> >> >
> >> >
> >> >On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com>
> wrote:
> >> >
> >> >> Hi,YI
> >> >>
> >> >> Thanks for putting together this FLIP — it's a well-thought-out
> >> proposal,
> >> >> and the RpcOperator Service is a really valuable addition for
> >> AI/inference
> >> >> workloads on Flink. I have a few questions and suggestions:
> >> >>
> >> >>
> >> >> 1. Autoscaling
> >> >> How is scale-up/scale-down expressed in this design? Could you
> consider
> >> >> allowing concurrency to be configured as a (min, max) range?
> Currently
> >> it
> >> >> appears only min = max (i.e. fixed parallelism) is supported.
> >> >>
> >> >>
> >> >> 2. Async invocation & end-to-end examples
> >> >> The example shows a synchronous invocation style. I'd suggest also
> >> >> providing an asynchronous invocation example — or even just a
> scaffold
> >> that
> >> >> exposes only the RpcOperator's handle and method. In addition, it
> would
> >> be
> >> >> best to include a complete end-to-end example of invoking a model
> (CPU
> >> >> inference would be fine).
> >> >>
> >> >>
> >> >> 3. GPU as a resource
> >> >> GPU is surely a common/first-class resource. Accessing it here
> through
> >> an
> >> >> extend mechanism — wouldn't that mislead users? It would seem more
> >> >> natural to declare GPU as a standard resource rather than going
> through
> >> an
> >> >> extension.
> >> >>
> >> >>
> >> >> 4. Serial method invocation
> >> >> Each method of an RpcOperator is invoked serially. I'd suggest
> stating
> >> >> this explicitly in both the documentation and the code comments, so
> that
> >> >> users — or an AI — can be clearly aware of it.
> >> >>
> >> >>
> >> >> 5. Why synchronous methods?
> >> >> Why is every method in RpcOperator a synchronous call? Does this
> imply
> >> >> that no model inference supports concurrent invocation?
> >> >>
> >> >>
> >> >> 6. SQL UDF usage
> >> >> How would SQL UDF users invoke this capability? I'd suggest
> providing a
> >> >> complete example for this as well.
> >> >>
> >> >>
> >> >> Thanks again for the great work!
> >> >>
> >> >> Best,
> >> >> Guowei
> >> >>
> >> >>
> >> >> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <
> dockerzhang@apache.org>
> >> >> wrote:
> >> >>
> >> >>> Hi Yi, and Flink Community,
> >> >>>
> >> >>> Thanks for bringing up this excellent proposal. I am fully in favor
> of
> >> >>> FLIP-582.
> >> >>>
> >> >>> In our production workloads, especially regarding large-scale AI
> >> >>> inference,
> >> >>> the tight coupling of the data plane and compute units has always
> been
> >> a
> >> >>> major pain point. If an inference subtask fails due to a GPU OOM or
> >> driver
> >> >>> issue, triggering a global rollback is incredibly expensive since
> model
> >> >>> reloading takes minutes.
> >> >>>
> >> >>> The introduction of the RpcOperator Service as a first-class
> primitive
> >> >>> masterfully decouples the heavy inference tasks from the mainstream
> >> >>> topology. The fault isolation, independent scaling, and stateless
> >> design
> >> >>> perfectly match the requirements of modern AI-oriented data
> processing.
> >> >>>
> >> >>> This is a clean and robust architecture. Looking forward to seeing
> this
> >> >>> merged!
> >> >>>
> >> >>>
> >> >>> Best wishes,
> >> >>> Charles Zhang
> >> >>> from Apache InLong
> >> >>>
> >> >>>
> >> >>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
> >> >>>
> >> >>> > Hi everyone,
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > I would like to start a discussion on FLIP-582: Support
> RpcOperator
> >> >>> > Service [1].
> >> >>> >
> >> >>> >
> >> >>> > AI-oriented workloads like multimodal data processing and model
> >> >>> inference
> >> >>> > are
> >> >>> > growing rapidly in recent years. These workloads are
> characterized by
> >> >>> > expensive
> >> >>> > resources (GPUs) and high initialization costs (seconds to minutes
> >> for
> >> >>> > model
> >> >>> > loading). In today's Flink, embedding them in the data plane
> couples
> >> >>> their
> >> >>> > parallelism and failover with surrounding operators; deploying
> them
> >> as
> >> >>> > external
> >> >>> > services disconnects their lifecycle from the job and doubles
> >> >>> operational
> >> >>> > overhead.
> >> >>> >
> >> >>> >
> >> >>> > This FLIP introduces RpcOperator Service — a framework-level
> >> primitive
> >> >>> > that runs
> >> >>> > user-defined compute as RPC services in an independent Pipelined
> >> Region
> >> >>> > within
> >> >>> > the Flink job. Because the service is isolated at the scheduling
> >> level,
> >> >>> it
> >> >>> > can achieve
> >> >>> > fault isolation, independent scaling, and dedicated resource
> >> allocation.
> >> >>> > As a native
> >> >>> > Flink primitive, it also lays the foundation for automatic flow
> >> control,
> >> >>> > flexible load
> >> >>> > balancing, and coordinated auto-scaling — all without introducing
> >> >>> external
> >> >>> > infrastructure or additional operational burden.
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > Looking forward to your feedback and suggestions!
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > [1]
> >> >>> >
> >> >>> >
> >> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > Best Regards,
> >> >>> > Yi Zhang
> >> >>>
> >> >>
> >>
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Gen,
Thanks for the thorough review and the insightful implementation-level questions!
Here are my responses to each point:
1. Heterogeneous resources
I fully agree that heterogeneous resource scheduling is a critical part of realizing
the full value of decoupled GPU/CPU scaling. However, as mentioned in my reply
to Hongshun's Q4, this is fundamentally a resource allocation and scheduling
problem that goes beyond the scope of this FLIP. We plan to address it in a
dedicated GPU and resource scheduling FLIP. The current design already uses
separate SlotSharingGroups for ROS services, and the interface is designed to
be extended without breaking changes when the resource scheduling FLIP lands.
2.1 RpcTask Design
This is a very thoughtful architectural suggestion, and I appreciate the detailed
reasoning from your StreamTask experience. You are right that the current FLIP
focuses more on the interfaces, and some implementation details like Mailbox
integration deserve further discussion.
The key reason for not introducing Mailbox in the current RpcTask is that the
RpcOperator does not yet have a strong enough need for it — especially given
the stateless design, which already covers a wide range of scenarios. The
simple queue + single-threaded run loop is sufficient to implement all currently
required capabilities. This keeps the implementation minimal and easy to
reason about.
That said, as the RpcOperator's capabilities expand in the future — such as
supporting checkpoint participation — introducing Mailbox would be a natural
and valuable evolution. Importantly, this is a purely internal refactoring that does
not affect any user-facing interfaces, so it can be adopted incrementally when
the need arises.
2.2. Communication implementation
This concern is similar to Xintong's question on the communication layer. The
first version directly reuses Flink's existing Pekko RPC infrastructure, but in the
longer term, introducing an alternative transport layer such as gRPC — which
natively supports streaming — would address both the large-payload concern
and the unfold/flatMap scenario you raised. This can be considered as part of
the communication layer evolution, and the upper-level `RosClient` /
`RpcOperator` interfaces are designed to be transport-agnostic.
2.3. Client-side implementation
This is indeed a key question. Let me clarify how the client-side checkpoint
interaction works. The `RosClient` provides an asynchronous interface
(`CompletableFuture`), and users have two usage patterns:
a. Synchronous usage (e.g., `client.request(...).get()` in a `ProcessFunction`):
The request completes before `processElement` returns, so there are no
in-flight requests at checkpoint time. Checkpoint is not a concern in this case.
b. Asynchronous usage: Note that users should use Flink's `AsyncDataStream`
with `RichAsyncFunction`, which already has built-in handling for pending
requests during checkpoints. The `RosClient` itself does not need to handle
any pending request recovery, just as any other async HTTP or RPC client
used within `AsyncFunction` does not.
In other words, the checkpoint semantics are not a new problem introduced
by ROS — they are handled by the existing `AsyncDataStream` framework,
which users are already expected to use for asynchronous invocation patterns.
I will update the FLIP to emphasize this point more clearly.
3. ROS as container, not service
This is a great direction to explore. In fact, the current `RpcOperator` interface
is flexible enough to support this container-style usage: a user could implement
`open()` to start an external service process and the `processRequest()` method
could simply proxy to the external service, or — as you suggest — be bypassed
entirely if the data plane communicates directly with the external service
endpoint. This is a viable and valuable pattern. One trade-off to note is that
some of ROS's built-in capabilities — such as framework-level load balancing
— would not be utilized in this mode, since traffic bypasses the RPC request
queue. Users would need to evaluate whether the external service provides
equivalent capabilities or whether the direct-access performance benefit
outweighs the loss of framework-level features.
Thank you for raising this — it validates an important use case that the current
architecture can accommodate.
4. Naming
There is indeed a naming inconsistency — `addRpcService` can be renamed
to `addRpcOperatorService` to align with the other interfaces. Thanks for
pointing this out.
I hope these responses address your concerns. Please feel free to follow up
with any further questions or suggestions.
Best,
Yi
At 2026-06-02 11:15:24, "Gen Luo" <lu...@gmail.com> wrote:
>Hi Yi,
>
>Thanks for this impressive proposal. Adding an RPC service deployment
>primitive can greatly expand Flink's capability scope and flexibility, with
>clear benefits for AI workloads and agent applications. It can also play a
>role in scenarios like state management or data lake compaction — relevant
>to but decoupled from the data pipeline. This is a very promising
>direction. That said, I still have some questions.
>
>1. My biggest concern is heterogeneous resources, echoing Hongshun's Q4.
>Flink today uses homogeneous TM resources sliced for different task
>requirements, but decoupling GPU/CPU resources and enabling flexible
>independent scaling is one of the core values of this FLIP — and the
>current resource allocation approach cannot support this. We likely need
>new implementations for the resource allocation component.
>
>2. Some components are described only at the idea level without
>implementation details. Hopefully these can be supplemented later or
>discussed in dedicated threads. A few worth discussing:
>
> - RpcTask: It's currently described as a simple single-threaded loop,
>with no plan to use Mailbox or OperatorChain. But if we want to support
>async, checkpoint, etc. later, introducing Mailbox would be valuable —
>handle the input queue as defaultAction, with async/CP/system messages as
>mails. This keeps semantics consistent with StreamTask and reduces
>maintenance burden. System events like FO-triggered request cancellation
>can also be prioritized via mailbox. OperatorChain is what
>SubtaskCheckpointCoordinator directly contacts, so having RpcOperator as
>the only node in the chain wouldn't add much complexity and lowers the cost
>when CP support comes.
>
> - Communication implementation: Will it reuse the Pekko + proxy
>mechanism? Pekko has risks with large message bodies (OOM, silently drop
>oversized frame, etc.). Since ROS is user-facing, we can't fully avoid
>this, and large-model scenarios often carry sizable context — needs careful
>handling. Also, the current design doesn't include table-function-style
>interfaces for unfolding data, so a large unfolded result has to be
>returned as a single response, inflating per-request payload size and
>amplifying the same risk.
>
> - Client-side implementation: Only the interface is described, no
>implementation details. Checkpoint is the prominent issue — since ROS is
>stateless by design, state recording falls to the client. Do we go with
>drain or record? Drain may take a long time; record on a keyed stream may
>cause request distribution confusion after parallelism changes.
>
>3. ROS as container, not service: Is this allowed? If the system itself has
>service capabilities — especially non-Java stacks (agents, LLMs, also
>Python remote functions) — forwarding requests through the Flink Java layer
>adds unnecessary overhead. In that case RpcOperator would only handle
>lifecycle, while the framework manages resource scheduling and service
>discovery without carrying business traffic.
>
>nit: addRpcService is inconsistent with other places that use RpcOperator /
>the ros keyword. Worth unifying.
>
>Best,
>Gen
>
>On Mon, Jun 1, 2026 at 2:02 PM Yi Zhang <zh...@163.com> wrote:
>
>> Hi Guowei,
>>
>>
>> Thanks for the thoughtful review and valuable feedback! Your questions are
>> insightful
>> and have directly driven refinements to the proposal. Here are my
>> responses to each
>> point:
>>
>>
>> 1. Autoscaling
>> Supporting a (min, max) concurrency range is a great idea. However,
>> auto-scaling
>> involves much more than just the API surface — it requires discussion
>> around
>> scaling policies, coordination with the scheduler, metric-driven triggers,
>> and more.
>> To keep this FLIP focused, I'd like to defer that to a follow-up. Fixed
>> parallelism is
>> a common and necessary capability on its own, and the first version will
>> support
>> that. The interfaces can be extended later to include auto-scaling support
>> without
>> breaking changes.
>>
>>
>> 2. Async invocation & end-to-end examples
>> This is a good point. The `RosClient` already provides an async interface,
>> so using
>> it with `AsyncDataStream` is indeed more natural and idiomatic. I will add
>> an
>> asynchronous invocation example in the proposal. Regarding a complete
>> end-to-end
>> inference example: at the Java API level, it will still be primarily
>> conceptual, since
>> practical model inference examples really need Python API support (e.g.,
>> loading a
>> PyTorch or ONNX model). I will refine the existing example to make the
>> overall
>> invocation chain as close to a real-world scenario as possible.
>>
>>
>> 3. GPU as a resource
>> This is indeed a key concern. The `extendedResource` API in the original
>> proposal
>> comes from Flink's existing resource model (introduced by FLIP-108).
>> However, I
>> agree that GPU deserves first-class treatment rather than being accessed
>> through a
>> generic extension mechanism. Promoting GPU to a first-class resource will
>> involve
>> changes to both the API and the scheduling layer, which we plan to address
>> in a
>> dedicated FLIP. For now, I will remove the `extendedResource` API from
>> `RosDescriptor` to keep this FLIP focused on the RPC service primitive
>> itself. The
>> resource declaration API can be extended once the GPU-specific FLIP is in
>> place.
>>
>>
>> 4. Serial method invocation
>> Agreed. I will make the serial invocation semantics more prominent — both
>> in the
>> FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
>>
>>
>> 5. Why synchronous methods?
>> This is a key question. The initial design chose synchronous methods for
>> simplicity
>> — users don't need to worry about thread safety, and concurrency is
>> achieved by
>> scaling out instances (parallelism). However, I recognize that
>> asynchronous
>> processing is a critical requirement for certain scenarios (e.g., when the
>> RPC service
>> performs expensive operations that can be parallelized internally). I am
>> planning to
>> extend the current design to support asynchronous method execution: if a
>> user's
>> method returns a `CompletableFuture`, the framework will recognize it and
>> wait for
>> the future to complete before sending the response, without blocking the
>> invoke
>> thread. This allows the invoke thread to continue dispatching subsequent
>> requests,
>> while the user controls internal concurrency. Importantly,
>> `processRequest` calls
>> remain serial — the async behavior is purely opt-in.
>>
>>
>> 6. SQL UDF usage
>> The current design is primarily oriented toward the DataStream API, as it
>> provides
>> the foundational building blocks. I agree that SQL and Table API support
>> is an
>> important aspect to consider as well. On the invocation side, both Table
>> API and
>> SQL UDFs share the same runtime path through `FunctionContext`, so
>> enabling
>> RPC service access from UDFs is straightforward — I will add a
>> `getRosClient`
>> method to `FunctionContext` and include that change in this FLIP.
>> The real challenge for SQL lies on the definition side: how to define and
>> register
>> an RPC service using pure SQL syntax, which would require SQL grammar
>> extensions and planner changes. I think that part would be better
>> addressed in
>> a dedicated follow-up FLIP to give it the attention it deserves.
>>
>>
>> 7. Service sharing across jobs & lifecycle
>> In this FLIP, the RPC service is defined as part of the job (via
>> `env.addRpcService()`
>> before `env.execute()`). It cannot be accessed across jobs, and it is
>> released when
>> the job reaches a terminal state. I believe your question is more relevant
>> to the
>> "Shared RPC Service" mentioned in the Follow-up Tasks section. While it's
>> out of
>> scope for this FLIP, let me briefly describe the envisioned design: a
>> shared RPC
>> service would run as an independent job within the same Application,
>> accessible by
>> all jobs in that Application. It would terminate when the Application
>> ends, or could be
>> stopped manually via a user-facing API.
>> The embedded (per-job) service model discussed in this FLIP is simpler to
>> manage
>> and well-suited for many streaming scenarios, which is why we focus on it
>> first.
>> Importantly, both the embedded and shared modes will reuse the same set of
>> deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`, etc.)
>> — the
>> difference is only in lifecycle management and visibility scope.
>>
>>
>>
>>
>> I hope the above addresses your questions. I will update the proposal to
>> reflect the
>> changes mentioned above. Please feel free to follow up if you have any
>> further
>> questions or suggestions.
>>
>>
>> Best,
>> Yi
>>
>>
>>
>> At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
>> >HI,YI
>> >
>> >One more question I'd like to raise is around how the service behaves when
>> >it's referenced by more than one job. If two jobs are submitted within the
>> >same application and both reference the same RpcOperator service, is that
>> >service shared across them, and what is the expected behavior in that
>> case?
>> >I'm also curious how this plays out for bounded (batch) jobs in
>> particular:
>> >if one job completes while the other is still running, what does the
>> >service lifecycle look like — is the service torn down as soon as the
>> first
>> >job finishes, and if so, how does that affect the job that's still
>> running?
>> >
>> >Best,
>> >Guowei
>> >
>> >
>> >On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
>> >
>> >> Hi,YI
>> >>
>> >> Thanks for putting together this FLIP — it's a well-thought-out
>> proposal,
>> >> and the RpcOperator Service is a really valuable addition for
>> AI/inference
>> >> workloads on Flink. I have a few questions and suggestions:
>> >>
>> >>
>> >> 1. Autoscaling
>> >> How is scale-up/scale-down expressed in this design? Could you consider
>> >> allowing concurrency to be configured as a (min, max) range? Currently
>> it
>> >> appears only min = max (i.e. fixed parallelism) is supported.
>> >>
>> >>
>> >> 2. Async invocation & end-to-end examples
>> >> The example shows a synchronous invocation style. I'd suggest also
>> >> providing an asynchronous invocation example — or even just a scaffold
>> that
>> >> exposes only the RpcOperator's handle and method. In addition, it would
>> be
>> >> best to include a complete end-to-end example of invoking a model (CPU
>> >> inference would be fine).
>> >>
>> >>
>> >> 3. GPU as a resource
>> >> GPU is surely a common/first-class resource. Accessing it here through
>> an
>> >> extend mechanism — wouldn't that mislead users? It would seem more
>> >> natural to declare GPU as a standard resource rather than going through
>> an
>> >> extension.
>> >>
>> >>
>> >> 4. Serial method invocation
>> >> Each method of an RpcOperator is invoked serially. I'd suggest stating
>> >> this explicitly in both the documentation and the code comments, so that
>> >> users — or an AI — can be clearly aware of it.
>> >>
>> >>
>> >> 5. Why synchronous methods?
>> >> Why is every method in RpcOperator a synchronous call? Does this imply
>> >> that no model inference supports concurrent invocation?
>> >>
>> >>
>> >> 6. SQL UDF usage
>> >> How would SQL UDF users invoke this capability? I'd suggest providing a
>> >> complete example for this as well.
>> >>
>> >>
>> >> Thanks again for the great work!
>> >>
>> >> Best,
>> >> Guowei
>> >>
>> >>
>> >> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
>> >> wrote:
>> >>
>> >>> Hi Yi, and Flink Community,
>> >>>
>> >>> Thanks for bringing up this excellent proposal. I am fully in favor of
>> >>> FLIP-582.
>> >>>
>> >>> In our production workloads, especially regarding large-scale AI
>> >>> inference,
>> >>> the tight coupling of the data plane and compute units has always been
>> a
>> >>> major pain point. If an inference subtask fails due to a GPU OOM or
>> driver
>> >>> issue, triggering a global rollback is incredibly expensive since model
>> >>> reloading takes minutes.
>> >>>
>> >>> The introduction of the RpcOperator Service as a first-class primitive
>> >>> masterfully decouples the heavy inference tasks from the mainstream
>> >>> topology. The fault isolation, independent scaling, and stateless
>> design
>> >>> perfectly match the requirements of modern AI-oriented data processing.
>> >>>
>> >>> This is a clean and robust architecture. Looking forward to seeing this
>> >>> merged!
>> >>>
>> >>>
>> >>> Best wishes,
>> >>> Charles Zhang
>> >>> from Apache InLong
>> >>>
>> >>>
>> >>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>> >>>
>> >>> > Hi everyone,
>> >>> >
>> >>> >
>> >>> >
>> >>> > I would like to start a discussion on FLIP-582: Support RpcOperator
>> >>> > Service [1].
>> >>> >
>> >>> >
>> >>> > AI-oriented workloads like multimodal data processing and model
>> >>> inference
>> >>> > are
>> >>> > growing rapidly in recent years. These workloads are characterized by
>> >>> > expensive
>> >>> > resources (GPUs) and high initialization costs (seconds to minutes
>> for
>> >>> > model
>> >>> > loading). In today's Flink, embedding them in the data plane couples
>> >>> their
>> >>> > parallelism and failover with surrounding operators; deploying them
>> as
>> >>> > external
>> >>> > services disconnects their lifecycle from the job and doubles
>> >>> operational
>> >>> > overhead.
>> >>> >
>> >>> >
>> >>> > This FLIP introduces RpcOperator Service — a framework-level
>> primitive
>> >>> > that runs
>> >>> > user-defined compute as RPC services in an independent Pipelined
>> Region
>> >>> > within
>> >>> > the Flink job. Because the service is isolated at the scheduling
>> level,
>> >>> it
>> >>> > can achieve
>> >>> > fault isolation, independent scaling, and dedicated resource
>> allocation.
>> >>> > As a native
>> >>> > Flink primitive, it also lays the foundation for automatic flow
>> control,
>> >>> > flexible load
>> >>> > balancing, and coordinated auto-scaling — all without introducing
>> >>> external
>> >>> > infrastructure or additional operational burden.
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > Looking forward to your feedback and suggestions!
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > [1]
>> >>> >
>> >>> >
>> >>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > Best Regards,
>> >>> > Yi Zhang
>> >>>
>> >>
>>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Gen Luo <lu...@gmail.com>.
Hi Yi,
Thanks for this impressive proposal. Adding an RPC service deployment
primitive can greatly expand Flink's capability scope and flexibility, with
clear benefits for AI workloads and agent applications. It can also play a
role in scenarios like state management or data lake compaction — relevant
to but decoupled from the data pipeline. This is a very promising
direction. That said, I still have some questions.
1. My biggest concern is heterogeneous resources, echoing Hongshun's Q4.
Flink today uses homogeneous TM resources sliced for different task
requirements, but decoupling GPU/CPU resources and enabling flexible
independent scaling is one of the core values of this FLIP — and the
current resource allocation approach cannot support this. We likely need
new implementations for the resource allocation component.
2. Some components are described only at the idea level without
implementation details. Hopefully these can be supplemented later or
discussed in dedicated threads. A few worth discussing:
- RpcTask: It's currently described as a simple single-threaded loop,
with no plan to use Mailbox or OperatorChain. But if we want to support
async, checkpoint, etc. later, introducing Mailbox would be valuable —
handle the input queue as defaultAction, with async/CP/system messages as
mails. This keeps semantics consistent with StreamTask and reduces
maintenance burden. System events like FO-triggered request cancellation
can also be prioritized via mailbox. OperatorChain is what
SubtaskCheckpointCoordinator directly contacts, so having RpcOperator as
the only node in the chain wouldn't add much complexity and lowers the cost
when CP support comes.
- Communication implementation: Will it reuse the Pekko + proxy
mechanism? Pekko has risks with large message bodies (OOM, silently drop
oversized frame, etc.). Since ROS is user-facing, we can't fully avoid
this, and large-model scenarios often carry sizable context — needs careful
handling. Also, the current design doesn't include table-function-style
interfaces for unfolding data, so a large unfolded result has to be
returned as a single response, inflating per-request payload size and
amplifying the same risk.
- Client-side implementation: Only the interface is described, no
implementation details. Checkpoint is the prominent issue — since ROS is
stateless by design, state recording falls to the client. Do we go with
drain or record? Drain may take a long time; record on a keyed stream may
cause request distribution confusion after parallelism changes.
3. ROS as container, not service: Is this allowed? If the system itself has
service capabilities — especially non-Java stacks (agents, LLMs, also
Python remote functions) — forwarding requests through the Flink Java layer
adds unnecessary overhead. In that case RpcOperator would only handle
lifecycle, while the framework manages resource scheduling and service
discovery without carrying business traffic.
nit: addRpcService is inconsistent with other places that use RpcOperator /
the ros keyword. Worth unifying.
Best,
Gen
On Mon, Jun 1, 2026 at 2:02 PM Yi Zhang <zh...@163.com> wrote:
> Hi Guowei,
>
>
> Thanks for the thoughtful review and valuable feedback! Your questions are
> insightful
> and have directly driven refinements to the proposal. Here are my
> responses to each
> point:
>
>
> 1. Autoscaling
> Supporting a (min, max) concurrency range is a great idea. However,
> auto-scaling
> involves much more than just the API surface — it requires discussion
> around
> scaling policies, coordination with the scheduler, metric-driven triggers,
> and more.
> To keep this FLIP focused, I'd like to defer that to a follow-up. Fixed
> parallelism is
> a common and necessary capability on its own, and the first version will
> support
> that. The interfaces can be extended later to include auto-scaling support
> without
> breaking changes.
>
>
> 2. Async invocation & end-to-end examples
> This is a good point. The `RosClient` already provides an async interface,
> so using
> it with `AsyncDataStream` is indeed more natural and idiomatic. I will add
> an
> asynchronous invocation example in the proposal. Regarding a complete
> end-to-end
> inference example: at the Java API level, it will still be primarily
> conceptual, since
> practical model inference examples really need Python API support (e.g.,
> loading a
> PyTorch or ONNX model). I will refine the existing example to make the
> overall
> invocation chain as close to a real-world scenario as possible.
>
>
> 3. GPU as a resource
> This is indeed a key concern. The `extendedResource` API in the original
> proposal
> comes from Flink's existing resource model (introduced by FLIP-108).
> However, I
> agree that GPU deserves first-class treatment rather than being accessed
> through a
> generic extension mechanism. Promoting GPU to a first-class resource will
> involve
> changes to both the API and the scheduling layer, which we plan to address
> in a
> dedicated FLIP. For now, I will remove the `extendedResource` API from
> `RosDescriptor` to keep this FLIP focused on the RPC service primitive
> itself. The
> resource declaration API can be extended once the GPU-specific FLIP is in
> place.
>
>
> 4. Serial method invocation
> Agreed. I will make the serial invocation semantics more prominent — both
> in the
> FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
>
>
> 5. Why synchronous methods?
> This is a key question. The initial design chose synchronous methods for
> simplicity
> — users don't need to worry about thread safety, and concurrency is
> achieved by
> scaling out instances (parallelism). However, I recognize that
> asynchronous
> processing is a critical requirement for certain scenarios (e.g., when the
> RPC service
> performs expensive operations that can be parallelized internally). I am
> planning to
> extend the current design to support asynchronous method execution: if a
> user's
> method returns a `CompletableFuture`, the framework will recognize it and
> wait for
> the future to complete before sending the response, without blocking the
> invoke
> thread. This allows the invoke thread to continue dispatching subsequent
> requests,
> while the user controls internal concurrency. Importantly,
> `processRequest` calls
> remain serial — the async behavior is purely opt-in.
>
>
> 6. SQL UDF usage
> The current design is primarily oriented toward the DataStream API, as it
> provides
> the foundational building blocks. I agree that SQL and Table API support
> is an
> important aspect to consider as well. On the invocation side, both Table
> API and
> SQL UDFs share the same runtime path through `FunctionContext`, so
> enabling
> RPC service access from UDFs is straightforward — I will add a
> `getRosClient`
> method to `FunctionContext` and include that change in this FLIP.
> The real challenge for SQL lies on the definition side: how to define and
> register
> an RPC service using pure SQL syntax, which would require SQL grammar
> extensions and planner changes. I think that part would be better
> addressed in
> a dedicated follow-up FLIP to give it the attention it deserves.
>
>
> 7. Service sharing across jobs & lifecycle
> In this FLIP, the RPC service is defined as part of the job (via
> `env.addRpcService()`
> before `env.execute()`). It cannot be accessed across jobs, and it is
> released when
> the job reaches a terminal state. I believe your question is more relevant
> to the
> "Shared RPC Service" mentioned in the Follow-up Tasks section. While it's
> out of
> scope for this FLIP, let me briefly describe the envisioned design: a
> shared RPC
> service would run as an independent job within the same Application,
> accessible by
> all jobs in that Application. It would terminate when the Application
> ends, or could be
> stopped manually via a user-facing API.
> The embedded (per-job) service model discussed in this FLIP is simpler to
> manage
> and well-suited for many streaming scenarios, which is why we focus on it
> first.
> Importantly, both the embedded and shared modes will reuse the same set of
> deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`, etc.)
> — the
> difference is only in lifecycle management and visibility scope.
>
>
>
>
> I hope the above addresses your questions. I will update the proposal to
> reflect the
> changes mentioned above. Please feel free to follow up if you have any
> further
> questions or suggestions.
>
>
> Best,
> Yi
>
>
>
> At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
> >HI,YI
> >
> >One more question I'd like to raise is around how the service behaves when
> >it's referenced by more than one job. If two jobs are submitted within the
> >same application and both reference the same RpcOperator service, is that
> >service shared across them, and what is the expected behavior in that
> case?
> >I'm also curious how this plays out for bounded (batch) jobs in
> particular:
> >if one job completes while the other is still running, what does the
> >service lifecycle look like — is the service torn down as soon as the
> first
> >job finishes, and if so, how does that affect the job that's still
> running?
> >
> >Best,
> >Guowei
> >
> >
> >On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
> >
> >> Hi,YI
> >>
> >> Thanks for putting together this FLIP — it's a well-thought-out
> proposal,
> >> and the RpcOperator Service is a really valuable addition for
> AI/inference
> >> workloads on Flink. I have a few questions and suggestions:
> >>
> >>
> >> 1. Autoscaling
> >> How is scale-up/scale-down expressed in this design? Could you consider
> >> allowing concurrency to be configured as a (min, max) range? Currently
> it
> >> appears only min = max (i.e. fixed parallelism) is supported.
> >>
> >>
> >> 2. Async invocation & end-to-end examples
> >> The example shows a synchronous invocation style. I'd suggest also
> >> providing an asynchronous invocation example — or even just a scaffold
> that
> >> exposes only the RpcOperator's handle and method. In addition, it would
> be
> >> best to include a complete end-to-end example of invoking a model (CPU
> >> inference would be fine).
> >>
> >>
> >> 3. GPU as a resource
> >> GPU is surely a common/first-class resource. Accessing it here through
> an
> >> extend mechanism — wouldn't that mislead users? It would seem more
> >> natural to declare GPU as a standard resource rather than going through
> an
> >> extension.
> >>
> >>
> >> 4. Serial method invocation
> >> Each method of an RpcOperator is invoked serially. I'd suggest stating
> >> this explicitly in both the documentation and the code comments, so that
> >> users — or an AI — can be clearly aware of it.
> >>
> >>
> >> 5. Why synchronous methods?
> >> Why is every method in RpcOperator a synchronous call? Does this imply
> >> that no model inference supports concurrent invocation?
> >>
> >>
> >> 6. SQL UDF usage
> >> How would SQL UDF users invoke this capability? I'd suggest providing a
> >> complete example for this as well.
> >>
> >>
> >> Thanks again for the great work!
> >>
> >> Best,
> >> Guowei
> >>
> >>
> >> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
> >> wrote:
> >>
> >>> Hi Yi, and Flink Community,
> >>>
> >>> Thanks for bringing up this excellent proposal. I am fully in favor of
> >>> FLIP-582.
> >>>
> >>> In our production workloads, especially regarding large-scale AI
> >>> inference,
> >>> the tight coupling of the data plane and compute units has always been
> a
> >>> major pain point. If an inference subtask fails due to a GPU OOM or
> driver
> >>> issue, triggering a global rollback is incredibly expensive since model
> >>> reloading takes minutes.
> >>>
> >>> The introduction of the RpcOperator Service as a first-class primitive
> >>> masterfully decouples the heavy inference tasks from the mainstream
> >>> topology. The fault isolation, independent scaling, and stateless
> design
> >>> perfectly match the requirements of modern AI-oriented data processing.
> >>>
> >>> This is a clean and robust architecture. Looking forward to seeing this
> >>> merged!
> >>>
> >>>
> >>> Best wishes,
> >>> Charles Zhang
> >>> from Apache InLong
> >>>
> >>>
> >>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
> >>>
> >>> > Hi everyone,
> >>> >
> >>> >
> >>> >
> >>> > I would like to start a discussion on FLIP-582: Support RpcOperator
> >>> > Service [1].
> >>> >
> >>> >
> >>> > AI-oriented workloads like multimodal data processing and model
> >>> inference
> >>> > are
> >>> > growing rapidly in recent years. These workloads are characterized by
> >>> > expensive
> >>> > resources (GPUs) and high initialization costs (seconds to minutes
> for
> >>> > model
> >>> > loading). In today's Flink, embedding them in the data plane couples
> >>> their
> >>> > parallelism and failover with surrounding operators; deploying them
> as
> >>> > external
> >>> > services disconnects their lifecycle from the job and doubles
> >>> operational
> >>> > overhead.
> >>> >
> >>> >
> >>> > This FLIP introduces RpcOperator Service — a framework-level
> primitive
> >>> > that runs
> >>> > user-defined compute as RPC services in an independent Pipelined
> Region
> >>> > within
> >>> > the Flink job. Because the service is isolated at the scheduling
> level,
> >>> it
> >>> > can achieve
> >>> > fault isolation, independent scaling, and dedicated resource
> allocation.
> >>> > As a native
> >>> > Flink primitive, it also lays the foundation for automatic flow
> control,
> >>> > flexible load
> >>> > balancing, and coordinated auto-scaling — all without introducing
> >>> external
> >>> > infrastructure or additional operational burden.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > Looking forward to your feedback and suggestions!
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > [1]
> >>> >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > Best Regards,
> >>> > Yi Zhang
> >>>
> >>
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Yi Zhang <zh...@163.com>.
Hi Guowei,
Thanks for the thoughtful review and valuable feedback! Your questions are insightful
and have directly driven refinements to the proposal. Here are my responses to each
point:
1. Autoscaling
Supporting a (min, max) concurrency range is a great idea. However, auto-scaling
involves much more than just the API surface — it requires discussion around
scaling policies, coordination with the scheduler, metric-driven triggers, and more.
To keep this FLIP focused, I'd like to defer that to a follow-up. Fixed parallelism is
a common and necessary capability on its own, and the first version will support
that. The interfaces can be extended later to include auto-scaling support without
breaking changes.
2. Async invocation & end-to-end examples
This is a good point. The `RosClient` already provides an async interface, so using
it with `AsyncDataStream` is indeed more natural and idiomatic. I will add an
asynchronous invocation example in the proposal. Regarding a complete end-to-end
inference example: at the Java API level, it will still be primarily conceptual, since
practical model inference examples really need Python API support (e.g., loading a
PyTorch or ONNX model). I will refine the existing example to make the overall
invocation chain as close to a real-world scenario as possible.
3. GPU as a resource
This is indeed a key concern. The `extendedResource` API in the original proposal
comes from Flink's existing resource model (introduced by FLIP-108). However, I
agree that GPU deserves first-class treatment rather than being accessed through a
generic extension mechanism. Promoting GPU to a first-class resource will involve
changes to both the API and the scheduling layer, which we plan to address in a
dedicated FLIP. For now, I will remove the `extendedResource` API from
`RosDescriptor` to keep this FLIP focused on the RPC service primitive itself. The
resource declaration API can be extended once the GPU-specific FLIP is in place.
4. Serial method invocation
Agreed. I will make the serial invocation semantics more prominent — both in the
FLIP documentation and in the code-level JavaDoc for `RpcOperator`.
5. Why synchronous methods?
This is a key question. The initial design chose synchronous methods for simplicity
— users don't need to worry about thread safety, and concurrency is achieved by
scaling out instances (parallelism). However, I recognize that asynchronous
processing is a critical requirement for certain scenarios (e.g., when the RPC service
performs expensive operations that can be parallelized internally). I am planning to
extend the current design to support asynchronous method execution: if a user's
method returns a `CompletableFuture`, the framework will recognize it and wait for
the future to complete before sending the response, without blocking the invoke
thread. This allows the invoke thread to continue dispatching subsequent requests,
while the user controls internal concurrency. Importantly, `processRequest` calls
remain serial — the async behavior is purely opt-in.
6. SQL UDF usage
The current design is primarily oriented toward the DataStream API, as it provides
the foundational building blocks. I agree that SQL and Table API support is an
important aspect to consider as well. On the invocation side, both Table API and
SQL UDFs share the same runtime path through `FunctionContext`, so enabling
RPC service access from UDFs is straightforward — I will add a `getRosClient`
method to `FunctionContext` and include that change in this FLIP.
The real challenge for SQL lies on the definition side: how to define and register
an RPC service using pure SQL syntax, which would require SQL grammar
extensions and planner changes. I think that part would be better addressed in
a dedicated follow-up FLIP to give it the attention it deserves.
7. Service sharing across jobs & lifecycle
In this FLIP, the RPC service is defined as part of the job (via `env.addRpcService()`
before `env.execute()`). It cannot be accessed across jobs, and it is released when
the job reaches a terminal state. I believe your question is more relevant to the
"Shared RPC Service" mentioned in the Follow-up Tasks section. While it's out of
scope for this FLIP, let me briefly describe the envisioned design: a shared RPC
service would run as an independent job within the same Application, accessible by
all jobs in that Application. It would terminate when the Application ends, or could be
stopped manually via a user-facing API.
The embedded (per-job) service model discussed in this FLIP is simpler to manage
and well-suited for many streaming scenarios, which is why we focus on it first.
Importantly, both the embedded and shared modes will reuse the same set of
deployment primitives (`RpcOperator`, `RosClient`, `RosDescriptor`, etc.) — the
difference is only in lifecycle management and visibility scope.
I hope the above addresses your questions. I will update the proposal to reflect the
changes mentioned above. Please feel free to follow up if you have any further
questions or suggestions.
Best,
Yi
At 2026-05-28 16:39:49, "Guowei Ma" <gu...@gmail.com> wrote:
>HI,YI
>
>One more question I'd like to raise is around how the service behaves when
>it's referenced by more than one job. If two jobs are submitted within the
>same application and both reference the same RpcOperator service, is that
>service shared across them, and what is the expected behavior in that case?
>I'm also curious how this plays out for bounded (batch) jobs in particular:
>if one job completes while the other is still running, what does the
>service lifecycle look like — is the service torn down as soon as the first
>job finishes, and if so, how does that affect the job that's still running?
>
>Best,
>Guowei
>
>
>On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
>
>> Hi,YI
>>
>> Thanks for putting together this FLIP — it's a well-thought-out proposal,
>> and the RpcOperator Service is a really valuable addition for AI/inference
>> workloads on Flink. I have a few questions and suggestions:
>>
>>
>> 1. Autoscaling
>> How is scale-up/scale-down expressed in this design? Could you consider
>> allowing concurrency to be configured as a (min, max) range? Currently it
>> appears only min = max (i.e. fixed parallelism) is supported.
>>
>>
>> 2. Async invocation & end-to-end examples
>> The example shows a synchronous invocation style. I'd suggest also
>> providing an asynchronous invocation example — or even just a scaffold that
>> exposes only the RpcOperator's handle and method. In addition, it would be
>> best to include a complete end-to-end example of invoking a model (CPU
>> inference would be fine).
>>
>>
>> 3. GPU as a resource
>> GPU is surely a common/first-class resource. Accessing it here through an
>> extend mechanism — wouldn't that mislead users? It would seem more
>> natural to declare GPU as a standard resource rather than going through an
>> extension.
>>
>>
>> 4. Serial method invocation
>> Each method of an RpcOperator is invoked serially. I'd suggest stating
>> this explicitly in both the documentation and the code comments, so that
>> users — or an AI — can be clearly aware of it.
>>
>>
>> 5. Why synchronous methods?
>> Why is every method in RpcOperator a synchronous call? Does this imply
>> that no model inference supports concurrent invocation?
>>
>>
>> 6. SQL UDF usage
>> How would SQL UDF users invoke this capability? I'd suggest providing a
>> complete example for this as well.
>>
>>
>> Thanks again for the great work!
>>
>> Best,
>> Guowei
>>
>>
>> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
>> wrote:
>>
>>> Hi Yi, and Flink Community,
>>>
>>> Thanks for bringing up this excellent proposal. I am fully in favor of
>>> FLIP-582.
>>>
>>> In our production workloads, especially regarding large-scale AI
>>> inference,
>>> the tight coupling of the data plane and compute units has always been a
>>> major pain point. If an inference subtask fails due to a GPU OOM or driver
>>> issue, triggering a global rollback is incredibly expensive since model
>>> reloading takes minutes.
>>>
>>> The introduction of the RpcOperator Service as a first-class primitive
>>> masterfully decouples the heavy inference tasks from the mainstream
>>> topology. The fault isolation, independent scaling, and stateless design
>>> perfectly match the requirements of modern AI-oriented data processing.
>>>
>>> This is a clean and robust architecture. Looking forward to seeing this
>>> merged!
>>>
>>>
>>> Best wishes,
>>> Charles Zhang
>>> from Apache InLong
>>>
>>>
>>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>>>
>>> > Hi everyone,
>>> >
>>> >
>>> >
>>> > I would like to start a discussion on FLIP-582: Support RpcOperator
>>> > Service [1].
>>> >
>>> >
>>> > AI-oriented workloads like multimodal data processing and model
>>> inference
>>> > are
>>> > growing rapidly in recent years. These workloads are characterized by
>>> > expensive
>>> > resources (GPUs) and high initialization costs (seconds to minutes for
>>> > model
>>> > loading). In today's Flink, embedding them in the data plane couples
>>> their
>>> > parallelism and failover with surrounding operators; deploying them as
>>> > external
>>> > services disconnects their lifecycle from the job and doubles
>>> operational
>>> > overhead.
>>> >
>>> >
>>> > This FLIP introduces RpcOperator Service — a framework-level primitive
>>> > that runs
>>> > user-defined compute as RPC services in an independent Pipelined Region
>>> > within
>>> > the Flink job. Because the service is isolated at the scheduling level,
>>> it
>>> > can achieve
>>> > fault isolation, independent scaling, and dedicated resource allocation.
>>> > As a native
>>> > Flink primitive, it also lays the foundation for automatic flow control,
>>> > flexible load
>>> > balancing, and coordinated auto-scaling — all without introducing
>>> external
>>> > infrastructure or additional operational burden.
>>> >
>>> >
>>> >
>>> >
>>> > Looking forward to your feedback and suggestions!
>>> >
>>> >
>>> >
>>> >
>>> > [1]
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Best Regards,
>>> > Yi Zhang
>>>
>>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Guowei Ma <gu...@gmail.com>.
HI,YI
One more question I'd like to raise is around how the service behaves when
it's referenced by more than one job. If two jobs are submitted within the
same application and both reference the same RpcOperator service, is that
service shared across them, and what is the expected behavior in that case?
I'm also curious how this plays out for bounded (batch) jobs in particular:
if one job completes while the other is still running, what does the
service lifecycle look like — is the service torn down as soon as the first
job finishes, and if so, how does that affect the job that's still running?
Best,
Guowei
On Thu, May 28, 2026 at 2:55 PM Guowei Ma <gu...@gmail.com> wrote:
> Hi,YI
>
> Thanks for putting together this FLIP — it's a well-thought-out proposal,
> and the RpcOperator Service is a really valuable addition for AI/inference
> workloads on Flink. I have a few questions and suggestions:
>
>
> 1. Autoscaling
> How is scale-up/scale-down expressed in this design? Could you consider
> allowing concurrency to be configured as a (min, max) range? Currently it
> appears only min = max (i.e. fixed parallelism) is supported.
>
>
> 2. Async invocation & end-to-end examples
> The example shows a synchronous invocation style. I'd suggest also
> providing an asynchronous invocation example — or even just a scaffold that
> exposes only the RpcOperator's handle and method. In addition, it would be
> best to include a complete end-to-end example of invoking a model (CPU
> inference would be fine).
>
>
> 3. GPU as a resource
> GPU is surely a common/first-class resource. Accessing it here through an
> extend mechanism — wouldn't that mislead users? It would seem more
> natural to declare GPU as a standard resource rather than going through an
> extension.
>
>
> 4. Serial method invocation
> Each method of an RpcOperator is invoked serially. I'd suggest stating
> this explicitly in both the documentation and the code comments, so that
> users — or an AI — can be clearly aware of it.
>
>
> 5. Why synchronous methods?
> Why is every method in RpcOperator a synchronous call? Does this imply
> that no model inference supports concurrent invocation?
>
>
> 6. SQL UDF usage
> How would SQL UDF users invoke this capability? I'd suggest providing a
> complete example for this as well.
>
>
> Thanks again for the great work!
>
> Best,
> Guowei
>
>
> On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
> wrote:
>
>> Hi Yi, and Flink Community,
>>
>> Thanks for bringing up this excellent proposal. I am fully in favor of
>> FLIP-582.
>>
>> In our production workloads, especially regarding large-scale AI
>> inference,
>> the tight coupling of the data plane and compute units has always been a
>> major pain point. If an inference subtask fails due to a GPU OOM or driver
>> issue, triggering a global rollback is incredibly expensive since model
>> reloading takes minutes.
>>
>> The introduction of the RpcOperator Service as a first-class primitive
>> masterfully decouples the heavy inference tasks from the mainstream
>> topology. The fault isolation, independent scaling, and stateless design
>> perfectly match the requirements of modern AI-oriented data processing.
>>
>> This is a clean and robust architecture. Looking forward to seeing this
>> merged!
>>
>>
>> Best wishes,
>> Charles Zhang
>> from Apache InLong
>>
>>
>> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>>
>> > Hi everyone,
>> >
>> >
>> >
>> > I would like to start a discussion on FLIP-582: Support RpcOperator
>> > Service [1].
>> >
>> >
>> > AI-oriented workloads like multimodal data processing and model
>> inference
>> > are
>> > growing rapidly in recent years. These workloads are characterized by
>> > expensive
>> > resources (GPUs) and high initialization costs (seconds to minutes for
>> > model
>> > loading). In today's Flink, embedding them in the data plane couples
>> their
>> > parallelism and failover with surrounding operators; deploying them as
>> > external
>> > services disconnects their lifecycle from the job and doubles
>> operational
>> > overhead.
>> >
>> >
>> > This FLIP introduces RpcOperator Service — a framework-level primitive
>> > that runs
>> > user-defined compute as RPC services in an independent Pipelined Region
>> > within
>> > the Flink job. Because the service is isolated at the scheduling level,
>> it
>> > can achieve
>> > fault isolation, independent scaling, and dedicated resource allocation.
>> > As a native
>> > Flink primitive, it also lays the foundation for automatic flow control,
>> > flexible load
>> > balancing, and coordinated auto-scaling — all without introducing
>> external
>> > infrastructure or additional operational burden.
>> >
>> >
>> >
>> >
>> > Looking forward to your feedback and suggestions!
>> >
>> >
>> >
>> >
>> > [1]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>> >
>> >
>> >
>> >
>> >
>> > Best Regards,
>> > Yi Zhang
>>
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Guowei Ma <gu...@gmail.com>.
Hi,YI
Thanks for putting together this FLIP — it's a well-thought-out proposal,
and the RpcOperator Service is a really valuable addition for AI/inference
workloads on Flink. I have a few questions and suggestions:
1. Autoscaling
How is scale-up/scale-down expressed in this design? Could you consider
allowing concurrency to be configured as a (min, max) range? Currently it
appears only min = max (i.e. fixed parallelism) is supported.
2. Async invocation & end-to-end examples
The example shows a synchronous invocation style. I'd suggest also
providing an asynchronous invocation example — or even just a scaffold that
exposes only the RpcOperator's handle and method. In addition, it would be
best to include a complete end-to-end example of invoking a model (CPU
inference would be fine).
3. GPU as a resource
GPU is surely a common/first-class resource. Accessing it here through an
extend mechanism — wouldn't that mislead users? It would seem more natural
to declare GPU as a standard resource rather than going through an
extension.
4. Serial method invocation
Each method of an RpcOperator is invoked serially. I'd suggest stating this
explicitly in both the documentation and the code comments, so that users —
or an AI — can be clearly aware of it.
5. Why synchronous methods?
Why is every method in RpcOperator a synchronous call? Does this imply that
no model inference supports concurrent invocation?
6. SQL UDF usage
How would SQL UDF users invoke this capability? I'd suggest providing a
complete example for this as well.
Thanks again for the great work!
Best,
Guowei
On Wed, May 27, 2026 at 8:36 PM Charles Zhang <do...@apache.org>
wrote:
> Hi Yi, and Flink Community,
>
> Thanks for bringing up this excellent proposal. I am fully in favor of
> FLIP-582.
>
> In our production workloads, especially regarding large-scale AI inference,
> the tight coupling of the data plane and compute units has always been a
> major pain point. If an inference subtask fails due to a GPU OOM or driver
> issue, triggering a global rollback is incredibly expensive since model
> reloading takes minutes.
>
> The introduction of the RpcOperator Service as a first-class primitive
> masterfully decouples the heavy inference tasks from the mainstream
> topology. The fault isolation, independent scaling, and stateless design
> perfectly match the requirements of modern AI-oriented data processing.
>
> This is a clean and robust architecture. Looking forward to seeing this
> merged!
>
>
> Best wishes,
> Charles Zhang
> from Apache InLong
>
>
> Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
>
> > Hi everyone,
> >
> >
> >
> > I would like to start a discussion on FLIP-582: Support RpcOperator
> > Service [1].
> >
> >
> > AI-oriented workloads like multimodal data processing and model inference
> > are
> > growing rapidly in recent years. These workloads are characterized by
> > expensive
> > resources (GPUs) and high initialization costs (seconds to minutes for
> > model
> > loading). In today's Flink, embedding them in the data plane couples
> their
> > parallelism and failover with surrounding operators; deploying them as
> > external
> > services disconnects their lifecycle from the job and doubles operational
> > overhead.
> >
> >
> > This FLIP introduces RpcOperator Service — a framework-level primitive
> > that runs
> > user-defined compute as RPC services in an independent Pipelined Region
> > within
> > the Flink job. Because the service is isolated at the scheduling level,
> it
> > can achieve
> > fault isolation, independent scaling, and dedicated resource allocation.
> > As a native
> > Flink primitive, it also lays the foundation for automatic flow control,
> > flexible load
> > balancing, and coordinated auto-scaling — all without introducing
> external
> > infrastructure or additional operational burden.
> >
> >
> >
> >
> > Looking forward to your feedback and suggestions!
> >
> >
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
> >
> >
> >
> >
> >
> > Best Regards,
> > Yi Zhang
>
Re: [DISCUSS] FLIP-582: Support RpcOperator Service
Posted by Charles Zhang <do...@apache.org>.
Hi Yi, and Flink Community,
Thanks for bringing up this excellent proposal. I am fully in favor of
FLIP-582.
In our production workloads, especially regarding large-scale AI inference,
the tight coupling of the data plane and compute units has always been a
major pain point. If an inference subtask fails due to a GPU OOM or driver
issue, triggering a global rollback is incredibly expensive since model
reloading takes minutes.
The introduction of the RpcOperator Service as a first-class primitive
masterfully decouples the heavy inference tasks from the mainstream
topology. The fault isolation, independent scaling, and stateless design
perfectly match the requirements of modern AI-oriented data processing.
This is a clean and robust architecture. Looking forward to seeing this
merged!
Best wishes,
Charles Zhang
from Apache InLong
Yi Zhang <zh...@163.com> 于2026年5月27日周三 14:12写道:
> Hi everyone,
>
>
>
> I would like to start a discussion on FLIP-582: Support RpcOperator
> Service [1].
>
>
> AI-oriented workloads like multimodal data processing and model inference
> are
> growing rapidly in recent years. These workloads are characterized by
> expensive
> resources (GPUs) and high initialization costs (seconds to minutes for
> model
> loading). In today's Flink, embedding them in the data plane couples their
> parallelism and failover with surrounding operators; deploying them as
> external
> services disconnects their lifecycle from the job and doubles operational
> overhead.
>
>
> This FLIP introduces RpcOperator Service — a framework-level primitive
> that runs
> user-defined compute as RPC services in an independent Pipelined Region
> within
> the Flink job. Because the service is isolated at the scheduling level, it
> can achieve
> fault isolation, independent scaling, and dedicated resource allocation.
> As a native
> Flink primitive, it also lays the foundation for automatic flow control,
> flexible load
> balancing, and coordinated auto-scaling — all without introducing external
> infrastructure or additional operational burden.
>
>
>
>
> Looking forward to your feedback and suggestions!
>
>
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-582%3A+Support+RpcOperator+Service
>
>
>
>
>
> Best Regards,
> Yi Zhang