You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fluss.apache.org by Hongshun Wang <lo...@gmail.com> on 2025/08/07 03:51:05 UTC
[DISCUSS] FIP-12: Server Dynamic Config
Hi devs,
I'd like to start a discussion about FIP-12: Server Dynamic Config[1].
Currently, any changes to the server.yaml configuration in a Fluss cluster
require a full restart of the cluster, which negatively impacts stability
and availability. To improve operational agility and reduce downtime, we
propose introducing dynamic configuration capabilities, enabling runtime
modification of key parameters—such as enabling/disabling lake-streaming
integration features or managing user accounts—without requiring service
interruption.
The POC[2] code is provided to enable lake format. You can try and give
some advice.
Best
Hongshun
[1]
https://cwiki.apache.org/confluence/display/FLUSS/FIP-12%3A+Server+Dynamic+Config
[2] https://github.com/loserwang1024/fluss/tree/poc-dymanic-config.
Re: [DISCUSS] FIP-12: Server Dynamic Config
Posted by Hongshun Wang <lo...@gmail.com>.
Hi yuxia,
Thanks for your thoughtful review, it does have many problems.
> I feel the `Tuple2<Set<String>, Set<String>>` is a little of obscure.
Now I add a class AllowedConfigs.
> The key is of string type, and the value is also of string type.
We have no other better type to carries all this option type.
> setString(String key, String value) to obtain newConfig
Will use merged the map,then transform with Configuration.fromMap whose
param is alse Map<String, String>.
> ServerReconfigurable need to handle this case itself and throw a
ConfigException
Yes, ServerReconfigurable#validate will simply verify the param format it
needs. For example, if we set datalake.format=unknown, exception will
thrown: com.alibaba.fluss.exception.ConfigException: Invalid configuration
for datalake format null: Could not parse value 'UNKNOW' for key
'datalake.format'.
> Why is config_source in PbDescribeConfigsResponseInfo optional?
Modified to required now.
> Moreover, I feel this field might be unnecessary since it's always
INITIAL_SERVER_CONFIG
When describing config, the initial options(from server.yaml) will return
as INITIAL_SERVER_CONFIG while dynamic options will return as
DYNAMIC_SERVER_CONFIG.
> For PbAlterConfigsRequestInfo, I think config_operation should also be
required?
done it.
> Could you please provide an example describing the data in zk?
Add it as
*{"version":1,"config":{"datalake.format":"value1","key2":null,"key3":"value3"}}*
Best,
Hongshun
On Thu, Aug 7, 2025 at 1:46 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Hi, Hongshun.
>
> Thanks for the FIP, few comments:
>
> 1: Regarding to method `Tuple2<Set<String>, Set<String>>
> allowedConfigs();`, Is it possible the returned type `Tuple2<Set<String>,
> Set<String>>` be a class to make it more self-ducumenting? Currently,
> I feel the `Tuple2<Set<String>, Set<String>>` is a little of obscure.
>
> 2: Regarding the parameter for users to modify configurations via
> alterConfigs(Collection<AlterConfigOp> configs):
> The key is of string type, and the value is also of string type. Do we
> directly use setString(String key, String value) to obtain newConfig? For
> enum types, if the value is not within the enum's range—for example, if a
> user sets datalake.format=unknown—then
> newConfig.getOptional(DATALAKE_FORMAT) will directly throw a
> java.lang.IllegalArgumentException. Does the implementation of
> ServerReconfigurable need to handle this case itself and throw a
> ConfigException? Other types of configuration items have similar issues.
>
> 3: Why is config_source in PbDescribeConfigsResponseInfo optional? Why
> isn't it also required? Moreover, I feel this field might be unnecessary
> since it's always INITIAL_SERVER_CONFIG? For PbAlterConfigsRequestInfo, I
> think config_operation should also be required?
>
> 4: Could you please provide an example describing the data in zk?
>
>
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "loserwang1024" <lo...@gmail.com>
> 收件人: "dev" <de...@fluss.apache.org>
> 发送时间: 星期四, 2025年 8 月 07日 上午 11:51:05
> 主题: [SPAM][DISCUSS] FIP-12: Server Dynamic Config
>
> Hi devs,
>
> I'd like to start a discussion about FIP-12: Server Dynamic Config[1].
> Currently, any changes to the server.yaml configuration in a Fluss cluster
> require a full restart of the cluster, which negatively impacts stability
> and availability. To improve operational agility and reduce downtime, we
> propose introducing dynamic configuration capabilities, enabling runtime
> modification of key parameters—such as enabling/disabling lake-streaming
> integration features or managing user accounts—without requiring service
> interruption.
>
> The POC[2] code is provided to enable lake format. You can try and give
> some advice.
>
> Best
> Hongshun
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-12%3A+Server+Dynamic+Config
> [2] https://github.com/loserwang1024/fluss/tree/poc-dymanic-config.
>
[DISCUSS] FIP-12: Server Dynamic Config
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, Hongshun.
Thanks for the FIP, few comments:
1: Regarding to method `Tuple2<Set<String>, Set<String>> allowedConfigs();`, Is it possible the returned type `Tuple2<Set<String>, Set<String>>` be a class to make it more self-ducumenting? Currently,
I feel the `Tuple2<Set<String>, Set<String>>` is a little of obscure.
2: Regarding the parameter for users to modify configurations via alterConfigs(Collection<AlterConfigOp> configs):
The key is of string type, and the value is also of string type. Do we directly use setString(String key, String value) to obtain newConfig? For enum types, if the value is not within the enum's range—for example, if a user sets datalake.format=unknown—then newConfig.getOptional(DATALAKE_FORMAT) will directly throw a java.lang.IllegalArgumentException. Does the implementation of ServerReconfigurable need to handle this case itself and throw a ConfigException? Other types of configuration items have similar issues.
3: Why is config_source in PbDescribeConfigsResponseInfo optional? Why isn't it also required? Moreover, I feel this field might be unnecessary since it's always INITIAL_SERVER_CONFIG? For PbAlterConfigsRequestInfo, I think config_operation should also be required?
4: Could you please provide an example describing the data in zk?
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "loserwang1024" <lo...@gmail.com>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期四, 2025年 8 月 07日 上午 11:51:05
主题: [SPAM][DISCUSS] FIP-12: Server Dynamic Config
Hi devs,
I'd like to start a discussion about FIP-12: Server Dynamic Config[1].
Currently, any changes to the server.yaml configuration in a Fluss cluster
require a full restart of the cluster, which negatively impacts stability
and availability. To improve operational agility and reduce downtime, we
propose introducing dynamic configuration capabilities, enabling runtime
modification of key parameters—such as enabling/disabling lake-streaming
integration features or managing user accounts—without requiring service
interruption.
The POC[2] code is provided to enable lake format. You can try and give
some advice.
Best
Hongshun
[1]
https://cwiki.apache.org/confluence/display/FLUSS/FIP-12%3A+Server+Dynamic+Config
[2] https://github.com/loserwang1024/fluss/tree/poc-dymanic-config.
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Junbo Wang <be...@gmail.com>.
Hi Hongshun,
Thanks for the good questions. I've added the detailed design of Fluss act-as to FIP-49, including how act-user-id maps to ACLs on the server side. The related issue is tracked in [1].
[1] https://github.com/apache/fluss/issues/3837
Best regards,
Junbo Wang
> On 7 Aug 2026, at 16:57, Hongshun Wang <lo...@gmail.com> wrote:
>
> Could we clarify this boundary in the FIP, for example:
>
> - service mode is intended for trusted or single-identity deployments, and
> all authenticated Gateway users share the service account's permissions;
> - user mode provides per-user Fluss authorization and requires act-as support
> from the Fluss cluster.
>
> This would also make the compatibility statement more precise: service mode
> works with existing clusters, while user mode requires the corresponding
> server capability.
Fwd: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Hongshun Wang <lo...@gmail.com>.
Hi jinbo,
Also, act-user-id is intended to bridge authentication/authorization across
systems. It would be helpful to include an example (eg, a diagram) showing
how it can be combined with ACL to achieve permission isolation — for
instance, how different act-user-id values map to different ACL rules on
the Fluss server side.
Best,
Hongshun
---------- Forwarded message ---------
From: Hongshun Wang <lo...@gmail.com>
Date: Fri, Aug 7, 2026 at 4:49 PM
Subject: Re: [DISCUSS] FIP-49: Fluss Gateway REST API
To: <de...@fluss.apache.org>
Hi junbo,
I like the idea of the REST gateway. I think supporting a REST gateway is
more meaningful than supporting DataFusion, because for SQL data
exploration we already have Flink SQL, StarRocks, and Spark, but the log
collection pipeline currently has no alternative.
By the way, I love the idea of act-user-id. Maybe you can also describe the
Fluss server-side authentication parameters in this FIP, such as
client.security.protocol and the related security.* and client.security.*
parameters.
Best,
Hongshun
On Fri, Aug 7, 2026 at 12:46 AM Junbo Wang <be...@gmail.com> wrote:
> Hi all,
>
> Thanks everyone for the great feedback and the active discussion so far —
> it's been really helpful in shaping the proposal.
>
> Following the design in FIP-49, I've put together a PoC [1], built on top
> of Giorgos's branch [2] — thanks Giorgos for getting this started. It's
> still a work in progress and mainly covers the REST write path. I've also
> updated FIP-49 based on this discussion.
>
> Given how the discussion has converged, I'd like to start the vote
> tomorrow. If anyone would like more time for a final review, just let me
> know.
>
>
>
> [1] https://github.com/beryllw/fluss/commits/fip-49-poc/
>
> [2] https://github.com/gstamatakis95/fluss/tree/feat/FIP-49
>
>
> Best regards,
> Junbo Wang
>
> > On 6 Aug 2026, at 17:27, Junbo Wang <be...@gmail.com> wrote:
> >
> > Hi Leonard and David,
> >
> > Thanks for raising this point, David. FIP-49 will focus on native Fluss
> writes and lightweight lookups. As for the backend, FIP-49 will keep it to
> a simple, lightweight implementation on top of fluss-rust, which we can
> evolve and refactor when we bring in the query path with FIP-32.
> >
> > And thanks, Leonard, for helping clarify the authorization contract
> boundary — I'll update FIP-49 accordingly.
> >
> >
> >
> > Best regards,
> > Junbo Wang
> >
> >> On 6 Aug 2026, at 11:38, Leonard Xu <xb...@gmail.com> wrote:
> >>
> >> Hi Junbo and all,
> >>
> >> Thanks for the detailed proposal and the follow-up explanations.
> >>
> >> I like the overall direction of keeping native REST operations directly
> on
> >> top of fluss-rust, while separating them from the future SQL/DataFusion
> path.
> >> The existing discussion has also covered the major runtime concerns
> well.
> >>
> >> I have just one clarification about the authorization contract.
> >>
> >> In service mode, all requests reach Fluss as the gateway service
> account, so
> >> Fluss cannot enforce per-user ACLs. In user mode, per-user authorization
> >> depends on the future act-as capability.
> >>
> >> Could we clarify this boundary in the FIP, for example:
> >>
> >> - service mode is intended for trusted or single-identity deployments,
> and
> >> all authenticated Gateway users share the service account's
> permissions;
> >> - user mode provides per-user Fluss authorization and requires act-as
> support
> >> from the Fluss cluster.
> >>
> >> This would also make the compatibility statement more precise: service
> mode
> >> works with existing clusters, while user mode requires the corresponding
> >> server capability.
> >>
> >> Other details such as batch partition grouping, memory admission and
> exact
> >> serialization handling can be refined during the OpenAPI and
> implementation
> >> reviews.
> >>
> >> Overall, +1 on the proposal and thanks for driving this!
> >>
> >> Best,
> >> Leonard
> >
>
>
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Hongshun Wang <lo...@gmail.com>.
Hi junbo,
I like the idea of the REST gateway. I think supporting a REST gateway is
more meaningful than supporting DataFusion, because for SQL data
exploration we already have Flink SQL, StarRocks, and Spark, but the log
collection pipeline currently has no alternative.
By the way, I love the idea of act-user-id. Maybe you can also describe the
Fluss server-side authentication parameters in this FIP, such as
client.security.protocol and the related security.* and client.security.*
parameters.
Best,
Hongshun
On Fri, Aug 7, 2026 at 12:46 AM Junbo Wang <be...@gmail.com> wrote:
> Hi all,
>
> Thanks everyone for the great feedback and the active discussion so far —
> it's been really helpful in shaping the proposal.
>
> Following the design in FIP-49, I've put together a PoC [1], built on top
> of Giorgos's branch [2] — thanks Giorgos for getting this started. It's
> still a work in progress and mainly covers the REST write path. I've also
> updated FIP-49 based on this discussion.
>
> Given how the discussion has converged, I'd like to start the vote
> tomorrow. If anyone would like more time for a final review, just let me
> know.
>
>
>
> [1] https://github.com/beryllw/fluss/commits/fip-49-poc/
>
> [2] https://github.com/gstamatakis95/fluss/tree/feat/FIP-49
>
>
> Best regards,
> Junbo Wang
>
> > On 6 Aug 2026, at 17:27, Junbo Wang <be...@gmail.com> wrote:
> >
> > Hi Leonard and David,
> >
> > Thanks for raising this point, David. FIP-49 will focus on native Fluss
> writes and lightweight lookups. As for the backend, FIP-49 will keep it to
> a simple, lightweight implementation on top of fluss-rust, which we can
> evolve and refactor when we bring in the query path with FIP-32.
> >
> > And thanks, Leonard, for helping clarify the authorization contract
> boundary — I'll update FIP-49 accordingly.
> >
> >
> >
> > Best regards,
> > Junbo Wang
> >
> >> On 6 Aug 2026, at 11:38, Leonard Xu <xb...@gmail.com> wrote:
> >>
> >> Hi Junbo and all,
> >>
> >> Thanks for the detailed proposal and the follow-up explanations.
> >>
> >> I like the overall direction of keeping native REST operations directly
> on
> >> top of fluss-rust, while separating them from the future SQL/DataFusion
> path.
> >> The existing discussion has also covered the major runtime concerns
> well.
> >>
> >> I have just one clarification about the authorization contract.
> >>
> >> In service mode, all requests reach Fluss as the gateway service
> account, so
> >> Fluss cannot enforce per-user ACLs. In user mode, per-user authorization
> >> depends on the future act-as capability.
> >>
> >> Could we clarify this boundary in the FIP, for example:
> >>
> >> - service mode is intended for trusted or single-identity deployments,
> and
> >> all authenticated Gateway users share the service account's
> permissions;
> >> - user mode provides per-user Fluss authorization and requires act-as
> support
> >> from the Fluss cluster.
> >>
> >> This would also make the compatibility statement more precise: service
> mode
> >> works with existing clusters, while user mode requires the corresponding
> >> server capability.
> >>
> >> Other details such as batch partition grouping, memory admission and
> exact
> >> serialization handling can be refined during the OpenAPI and
> implementation
> >> reviews.
> >>
> >> Overall, +1 on the proposal and thanks for driving this!
> >>
> >> Best,
> >> Leonard
> >
>
>
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Junbo Wang <be...@gmail.com>.
Hi all,
Thanks everyone for the great feedback and the active discussion so far — it's been really helpful in shaping the proposal.
Following the design in FIP-49, I've put together a PoC [1], built on top of Giorgos's branch [2] — thanks Giorgos for getting this started. It's still a work in progress and mainly covers the REST write path. I've also updated FIP-49 based on this discussion.
Given how the discussion has converged, I'd like to start the vote tomorrow. If anyone would like more time for a final review, just let me know.
[1] https://github.com/beryllw/fluss/commits/fip-49-poc/
[2] https://github.com/gstamatakis95/fluss/tree/feat/FIP-49
Best regards,
Junbo Wang
> On 6 Aug 2026, at 17:27, Junbo Wang <be...@gmail.com> wrote:
>
> Hi Leonard and David,
>
> Thanks for raising this point, David. FIP-49 will focus on native Fluss writes and lightweight lookups. As for the backend, FIP-49 will keep it to a simple, lightweight implementation on top of fluss-rust, which we can evolve and refactor when we bring in the query path with FIP-32.
>
> And thanks, Leonard, for helping clarify the authorization contract boundary — I'll update FIP-49 accordingly.
>
>
>
> Best regards,
> Junbo Wang
>
>> On 6 Aug 2026, at 11:38, Leonard Xu <xb...@gmail.com> wrote:
>>
>> Hi Junbo and all,
>>
>> Thanks for the detailed proposal and the follow-up explanations.
>>
>> I like the overall direction of keeping native REST operations directly on
>> top of fluss-rust, while separating them from the future SQL/DataFusion path.
>> The existing discussion has also covered the major runtime concerns well.
>>
>> I have just one clarification about the authorization contract.
>>
>> In service mode, all requests reach Fluss as the gateway service account, so
>> Fluss cannot enforce per-user ACLs. In user mode, per-user authorization
>> depends on the future act-as capability.
>>
>> Could we clarify this boundary in the FIP, for example:
>>
>> - service mode is intended for trusted or single-identity deployments, and
>> all authenticated Gateway users share the service account's permissions;
>> - user mode provides per-user Fluss authorization and requires act-as support
>> from the Fluss cluster.
>>
>> This would also make the compatibility statement more precise: service mode
>> works with existing clusters, while user mode requires the corresponding
>> server capability.
>>
>> Other details such as batch partition grouping, memory admission and exact
>> serialization handling can be refined during the OpenAPI and implementation
>> reviews.
>>
>> Overall, +1 on the proposal and thanks for driving this!
>>
>> Best,
>> Leonard
>
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Junbo Wang <be...@gmail.com>.
Hi Leonard and David,
Thanks for raising this point, David. FIP-49 will focus on native Fluss writes and lightweight lookups. As for the backend, FIP-49 will keep it to a simple, lightweight implementation on top of fluss-rust, which we can evolve and refactor when we bring in the query path with FIP-32.
And thanks, Leonard, for helping clarify the authorization contract boundary — I'll update FIP-49 accordingly.
Best regards,
Junbo Wang
> On 6 Aug 2026, at 11:38, Leonard Xu <xb...@gmail.com> wrote:
>
> Hi Junbo and all,
>
> Thanks for the detailed proposal and the follow-up explanations.
>
> I like the overall direction of keeping native REST operations directly on
> top of fluss-rust, while separating them from the future SQL/DataFusion path.
> The existing discussion has also covered the major runtime concerns well.
>
> I have just one clarification about the authorization contract.
>
> In service mode, all requests reach Fluss as the gateway service account, so
> Fluss cannot enforce per-user ACLs. In user mode, per-user authorization
> depends on the future act-as capability.
>
> Could we clarify this boundary in the FIP, for example:
>
> - service mode is intended for trusted or single-identity deployments, and
> all authenticated Gateway users share the service account's permissions;
> - user mode provides per-user Fluss authorization and requires act-as support
> from the Fluss cluster.
>
> This would also make the compatibility statement more precise: service mode
> works with existing clusters, while user mode requires the corresponding
> server capability.
>
> Other details such as batch partition grouping, memory admission and exact
> serialization handling can be refined during the OpenAPI and implementation
> reviews.
>
> Overall, +1 on the proposal and thanks for driving this!
>
> Best,
> Leonard
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Leonard Xu <xb...@gmail.com>.
Hi Junbo and all,
Thanks for the detailed proposal and the follow-up explanations.
I like the overall direction of keeping native REST operations directly on
top of fluss-rust, while separating them from the future SQL/DataFusion path.
The existing discussion has also covered the major runtime concerns well.
I have just one clarification about the authorization contract.
In service mode, all requests reach Fluss as the gateway service account, so
Fluss cannot enforce per-user ACLs. In user mode, per-user authorization
depends on the future act-as capability.
Could we clarify this boundary in the FIP, for example:
- service mode is intended for trusted or single-identity deployments, and
all authenticated Gateway users share the service account's permissions;
- user mode provides per-user Fluss authorization and requires act-as support
from the Fluss cluster.
This would also make the compatibility statement more precise: service mode
works with existing clusters, while user mode requires the corresponding
server capability.
Other details such as batch partition grouping, memory admission and exact
serialization handling can be refined during the OpenAPI and implementation
reviews.
Overall, +1 on the proposal and thanks for driving this!
Best,
Leonard
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Junbo Wang <be...@gmail.com>.
Thanks Anton for the deep review — great catches, all points are now updated in the FIP-49.
1. the new cooperative backpressure / request timeout on writes
Both need client-side capabilities that fluss-rust doesn't have yet. I've filed issues to implement them — #3855 <https://github.com/apache/fluss/issues/3855>(cooperative backpressure, client side of #3463) and #3861 <https://github.com/apache/fluss/issues/3861> (a delivery.timeout.ms-style delivery bound) — under the umbrella #3854 <https://github.com/apache/fluss/issues/3854>, and the REST gateway will build on these capabilities.On top of them, the FIP-49 will run every write under an end-to-end deadline (gateway.rest.write.request-timeout, default 30s), with client delivery bounded below it. Pressure shows up as bounded latency, observable could via re-exported client metrics; hard rejections surface as a retriable entry-level storage_backpressure, so callers retry just those entries. At the deadline, in-flight entries are reported timeout (indeterminate); never-sent entries fail deterministically.
2. connections in service mode / per-user authorization
The FIP-49 design is simply two identity modes.
service: one shared connection per cluster — so you're right, connection.max can't fire there; it and idle-timeout are scoped to user mode only.
user: multiple connections per cluster, one per user id, using impersonation — the connection authenticates as a super-user and carries the end user as the SASL/PLAIN authorization id, so Fluss authorizes each request as that user against its own ACLs. That is the per-user authorization path; prerequisite is #3857 <https://github.com/apache/fluss/issues/3837>, and user mode fails at startup until it's integrated.
4.which tables get per-row results?
That was a wording error in the FIP — the REST surface is JSON-only and all writes are row-by-row, so every table gets per-row results. Corrected: the append_arrow_batch path is dropped from this FIP (deferred to a future columnar content type); the accumulator still batches per bucket, so nothing is lost.
5.duplicates on client retry
The FIP-49 design is at-least-once — HTTP retries are not deduplicated. KV tables converge per key; for log tables dedup belongs downstream (and retry only the entries in failures). Same stance as Kafka REST Proxy.
6. type names
Good catch — that was an oversight. Fixed: the FIP now uses the DataTypeJsonSerde vocabulary verbatim (INTEGER,TIMESTAMP_WITH_LOCAL_TIME_ZONE, ...), so REST schema objects are identical to the persisted serialization and no mapping exists anywhere.
All changes are updated on the Confluence page.Thanks again Anton for the thorough review — these suggestions were really helpful.
Best regards,
Junbo Wang
> On 4 Aug 2026, at 14:04, Junbo Wang <be...@gmail.com> wrote:
>
> Thanks for the MKO's advice — good point! We started with Prometheus pull for the MVP mainly for simplicity — it works out of the box and aligns with how Fluss servers expose metrics today. That said, it doesn't preclude OTel: a Collector can scrape the /metrics endpoint and re-export via OTLP, and we plan to keep the config extensible so a native OTLP exporter can be added later.
>
>
> Thanks for the great questions, Liting Liu.
>
> > Why Rust instead of Java/Netty?
> Three reasons: (a) the gateway is a proxy-style component where tail latency and footprint matter — Rust gives GC-free predictable latency and a much smaller memory footprint as a single static binary; (b) ecosystem fit — Fluss data is Arrow-native, and the Rust stack (arrow-rs, DataFusion, pgwire) keeps a zero-copy path end to end; (c) the gateway also helps drive the fluss-rust client to production readiness and grow the Rust ecosystem.
>
> > a query failure or OOM could potentially affect the entire Gateway.
>
> Query execution will run on a dedicated tokio runtime with a bounded thread pool, so it can't starve the write path. For memory, DataFusion's built-in MemoryPool sets per-query budgets — an over-limit query fails alone, not the process. We'll also add concurrency limits and query timeouts. If heavy workloads ever emerge, a separate query worker remains an option.
>
>
>
> > a lightweight lookup REST API backed directly by the Fluss client,
>
> Agreed — the initial read path is a lightweight lookup API backed directly by the Fluss client, no DataFusion. DataFusion comes later during FIP-32 implementation, and only for lightweight queries (filter/projection/limit).
>
>
> --
> Best regards,
> Junbo Wang
>
>
> Anton Borisov <borzoniusy@gmail.com <ma...@gmail.com>> 于2026年8月4日周二 10:30写道：
>> Hi Junbo,
>>
>> Nice proposal, thank you for driving it!
>> I like the direction overall, +1
>>
>> My questions below:
>>
>> 1. Have you seen the new cooperative backpressure?
>> #3463 [1] went in on 31 July, a day after this thread started. Tablet
>> servers now
>> send a per-bucket pressure value on PutKv responses, the client throttles on
>> that by up to 3 seconds by default, and the server can reject with a retriable
>> StorageBackpressureException when RocksDB is close to stalling.
>>
>> Inside an HTTP request that becomes invisible latency, the caller waits and
>> isn't told why. It also looks like the thing your rate limiting wants to be
>> driven by. A 429 with Retry-After says "slow down" in a way HTTP clients
>> already understand, and a fixed requests/sec limit can't track what the
>> cluster is actually doing. Worth pulling into the FIP?
>>
>> 2. What is a "connection" in service mode, and how many are there?
>> The page says service mode uses "one shared pool per cluster", and
>> connection.max is a "total connection cap (exceeded -> 429)". But it doesn't
>> say what a connection is, or how many service mode opens.
>>
>> It matters because a Fluss Connection isn't a socket. It's documented as
>> heavy-weight to create, thread-safe and meant to be shared, and each one
>> carries a 64 MB write buffer, a sender task, a writer id and sockets to every
>> tablet server. If service mode really is one shared connection, then the 512
>> cap and its 429 never fire, and it'd be clearer to say so. If it's many,
>> they'll batch worse than one would, since each accumulator sees only a
>> fraction of the traffic. Kafka REST Proxy shares a single producer across
>> requests for that reason, and only goes per-principal when identities differ.
>>
>> 3. Is there a request timeout on writes?
>> The client retries forever by default, and "leader not available" is one of
>> the errors it retries. So the row in your example wouldn't fail, it would
>> wait. And if the gateway stops waiting and reports a failure, does the row
>> stay queued and land later? That would make "failed" sometimes mean
>> "succeeded, just later".
>>
>> 4. Which tables get per-row results?
>> For Log tables without a bucket key you use append_arrow_batch, which returns
>> one result for the whole batch and stops at the first failure. I don't think
>> per-row reporting is possible on that path. Is it meant to be all-or-nothing
>> there?
>>
>> 5. What stops a duplicate when a client retries?
>> Our idempotence catches the writer resending its own batch. A retried HTTP
>> request is a different thing - the rows arrive fresh, so nothing links them to
>> the first attempt. The writer id also belongs to one process, so a second
>> gateway can't pick up where the first left off.
>>
>> 6. Is per-user authorization on the roadmap?
>> The page says Fluss-side ACLs see only the gateway account, so every request
>> reaches the cluster as gateway_svc and the server can't tell one caller from
>> another. Is something planned for later: which principals may write to which
>> tables, or is the gateway meant only for deployments where one shared
>> identity is fine?
>>
>> Last one, the type names. Fluss spells types two ways: INT and TIMESTAMP_LTZ
>> in SQL, INTEGER and TIMESTAMP_WITH_LOCAL_TIME_ZONE in DataTypeJsonSerde. The
>> FIP uses the SQL names with the serde's JSON shape, so I don't think the
>> examples parse as written.
>>
>> Which did you intend? If it's the SQL names, the gateway has to convert them
>> before calling CreateTable, and that mapping is worth writing into the FIP.
>>
>> Thank you!
>>
>> -- Anton
>>
>> [1] https://github.com/apache/fluss/pull/3463
>>
>> пн, 3 авг. 2026 г. в 16:02, David Reger <davidreger1337@gmail.com <ma...@gmail.com>>:
>> >
>> > I think this also relates to one of the architectural discussions we've
>> > recently had around FIP-32 [1].
>> >
>> > My understanding is that there is a useful distinction between *native
>> > Fluss operations* and *SQL/query execution*.
>> >
>> > For example, writes, metadata operations, and potentially lightweight
>> > native reads (such as point lookups) could continue to build directly on
>> > the fluss-rust foundation, whereas DataFusion would primarily serve the
>> > SQL/query path. This would allow the REST gateway to remain lightweight
>> > while keeping the query layer reusable for SQL-based protocols in the
>> > future.
>> >
>> > I don't think this necessarily changes the scope of FIP-49, but I do think
>> > it's an important architectural distinction as we evolve the gateway beyond
>> > the initial REST write MVP.
>> >
>> > [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
>> >
>> > Best regards,
>> >
>> > David
>> >
>> > Am Fr., 31. Juli 2026 um 05:02 Uhr schrieb Liting Liu (litiliu) via dev <
>> > dev@fluss.apache.org <ma...@fluss.apache.org>>:
>> >
>> > > Hi， Junbo
>> > >
>> > >
>> > > Thanks for the detailed proposal. I have a couple of questions about the
>> > > technology choice and the future SQL path.
>> > >
>> > > For the current scope—REST writes and metadata/DDL—have we considered
>> > > implementing the Gateway with a Java/Netty REST stack backed by the
>> > > existing Fluss Java client? This would be similar in spirit to how Flink
>> > > SQL Gateway reuses Flink’s native Java stack and Netty-based REST
>> > > infrastructure. The Fluss Java client already supports append, upsert,
>> > > delete, partial update, metadata operations, and lookup. It would be
>> > > helpful if the FIP could compare this alternative with the proposed
>> > >
>> > > I also have a question about the future DataFusion integration. Is
>> > > DataFusion expected to execute SQL queries inside the same Gateway process?
>> > > If so, complex or resource-intensive queries could compete with REST
>> > > ingestion and metadata operations for CPU and memory, and a query failure
>> > > or OOM could potentially affect the entire Gateway. Do we plan to provide
>> > > admission control, memory and concurrency limits, query
>> > > timeout/cancellation, spilling, or process-level isolation?
>> > >
>> > > If the initial read use case is mainly primary-key or prefix lookup, could
>> > > we expose a lightweight lookup REST API backed directly by the Fluss
>> > > client, without introducing DataFusion? DataFusion seems more valuable when
>> > > the intended scope includes general SQL capabilities such as scans,
>> > > filters, aggregations, and joins? If that is the long-term goal, would it
>> > > be preferable to keep complex SQL execution in a separate optional SQL
>> > > service or worker process, so that it does not increase the failure domain
>> > > of the REST write Gateway?
>> > >
>> > >
>> > > From: Michael Koepf <michael.koepf@outlook.com <ma...@outlook.com>>
>> > > Date: Thursday, July 30, 2026 at 22:15
>> > > To: dev@fluss.apache.org <ma...@fluss.apache.org> <dev@fluss.apache.org <ma...@fluss.apache.org>>
>> > > Subject: Re: [DISCUSS] FIP-49: Fluss Gateway REST API
>> > >
>> > > Hi Juno,
>> > >
>> > > Thanks for the FIP, great initiative.
>> > >
>> > > A question that comes to my mind: Why not expose metrics via vendor
>> > > neutral OTEL instead of limiting it to Prometheus?
>> > > --
>> > > Best,
>> > > MKO
>> > >
>> > >
>> > > > On 30.07.2026, at 12:23, Forward Xu <forwardxu315@gmail.com <ma...@gmail.com>> wrote:
>> > > >
>> > > > Thanks junbo,
>> > > >
>> > > > The proposal looks good to me in general. +1
>> > > >
>> > > > Best,
>> > > > Foward
>> > > >
>> > > > Junbo Wang <beryllwang@gmail.com <ma...@gmail.com>> 于2026年7月30日周四 11:49写道：
>> > > >
>> > > >> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST
>> > > API.
>> > > >>
>> > > >> The proposal is available here:
>> > > >>
>> > > https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API
>> > > >>
>> > > >> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] and
>> > > >> does not conflict with it. Following the earlier suggestion to split the
>> > > >> gateway work into focused FIPs, FIP-49 defines the REST write path and
>> > > DDL
>> > > >> / metadata management in detail, while REST read and the SQL path
>> > > remain in
>> > > >> the scope of FIP-32.
>> > > >>
>> > > >> In short, it proposes a standalone, stateless fluss-gateway process
>> > > (Rust,
>> > > >> axum + tokio, built on fluss-rust) providing:
>> > > >>
>> > > >> HTTP/JSON ingestion into KV and Log tables, at-least-once, with per-row
>> > > >> success/failure reporting
>> > > >> Full database / table / partition DDL
>> > > >> Pluggable client authentication, and a service / user (act-as) identity
>> > > >> mode towards Fluss
>> > > >> Multi-cluster routing, Prometheus metrics, a unified error model
>> > > >>
>> > > >> No Fluss server-side changes are required— the gateway accesses existing
>> > > >> clusters purely as a fluss-rust client.
>> > > >>
>> > > >> Looking forward to your feedback and suggestions.
>> > > >>
>> > > >>
>> > > >>
>> > > >> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
>> > > >>
>> > > >>
>> > > >> Best regards,
>> > > >> Junbo Wang
>> > > >>
>> > > >>
>> > >
>
>
>
>
>
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Junbo Wang <be...@gmail.com>.
Thanks for the MKO's advice — good point!
We started with Prometheus pull for the MVP mainly for simplicity — it
works out of the box and aligns with how Fluss servers expose metrics
today. That said, it doesn't preclude OTel: a Collector can scrape the
/metrics endpoint and re-export via OTLP, and we plan to keep the
config extensible so a native OTLP exporter can be added later.
Thanks for the great questions, Liting Liu.
> Why Rust instead of Java/Netty?
Three reasons: (a) the gateway is a proxy-style component where tail
latency and footprint matter — Rust gives GC-free predictable latency
and a much smaller memory footprint as a single static binary; (b)
ecosystem fit — Fluss data is Arrow-native, and the Rust stack
(arrow-rs, DataFusion, pgwire) keeps a zero-copy path end to end; (c)
the gateway also helps drive the fluss-rust client to production
readiness and grow the Rust ecosystem.
> a query failure or OOM could potentially affect the entire Gateway.
Query execution will run on a dedicated tokio runtime with a bounded
thread pool, so it can't starve the write path. For memory,
DataFusion's built-in MemoryPool sets per-query budgets — an
over-limit query fails alone, not the process. We'll also add
concurrency limits and query timeouts. If heavy workloads ever emerge,
a separate query worker remains an option.
> a lightweight lookup REST API backed directly by the Fluss client,
Agreed — the initial read path is a lightweight lookup API backed
directly by the Fluss client, no DataFusion. DataFusion comes later
during FIP-32 implementation, and only for lightweight queries
(filter/projection/limit).
--
Best regards,
Junbo Wang
Anton Borisov <bo...@gmail.com> 于2026年8月4日周二 10:30写道：
> Hi Junbo,
>
> Nice proposal, thank you for driving it!
> I like the direction overall, +1
>
> My questions below:
>
> 1. Have you seen the new cooperative backpressure?
> #3463 [1] went in on 31 July, a day after this thread started. Tablet
> servers now
> send a per-bucket pressure value on PutKv responses, the client throttles
> on
> that by up to 3 seconds by default, and the server can reject with a
> retriable
> StorageBackpressureException when RocksDB is close to stalling.
>
> Inside an HTTP request that becomes invisible latency, the caller waits and
> isn't told why. It also looks like the thing your rate limiting wants to be
> driven by. A 429 with Retry-After says "slow down" in a way HTTP clients
> already understand, and a fixed requests/sec limit can't track what the
> cluster is actually doing. Worth pulling into the FIP?
>
> 2. What is a "connection" in service mode, and how many are there?
> The page says service mode uses "one shared pool per cluster", and
> connection.max is a "total connection cap (exceeded -> 429)". But it
> doesn't
> say what a connection is, or how many service mode opens.
>
> It matters because a Fluss Connection isn't a socket. It's documented as
> heavy-weight to create, thread-safe and meant to be shared, and each one
> carries a 64 MB write buffer, a sender task, a writer id and sockets to
> every
> tablet server. If service mode really is one shared connection, then the
> 512
> cap and its 429 never fire, and it'd be clearer to say so. If it's many,
> they'll batch worse than one would, since each accumulator sees only a
> fraction of the traffic. Kafka REST Proxy shares a single producer across
> requests for that reason, and only goes per-principal when identities
> differ.
>
> 3. Is there a request timeout on writes?
> The client retries forever by default, and "leader not available" is one of
> the errors it retries. So the row in your example wouldn't fail, it would
> wait. And if the gateway stops waiting and reports a failure, does the row
> stay queued and land later? That would make "failed" sometimes mean
> "succeeded, just later".
>
> 4. Which tables get per-row results?
> For Log tables without a bucket key you use append_arrow_batch, which
> returns
> one result for the whole batch and stops at the first failure. I don't
> think
> per-row reporting is possible on that path. Is it meant to be
> all-or-nothing
> there?
>
> 5. What stops a duplicate when a client retries?
> Our idempotence catches the writer resending its own batch. A retried HTTP
> request is a different thing - the rows arrive fresh, so nothing links
> them to
> the first attempt. The writer id also belongs to one process, so a second
> gateway can't pick up where the first left off.
>
> 6. Is per-user authorization on the roadmap?
> The page says Fluss-side ACLs see only the gateway account, so every
> request
> reaches the cluster as gateway_svc and the server can't tell one caller
> from
> another. Is something planned for later: which principals may write to
> which
> tables, or is the gateway meant only for deployments where one shared
> identity is fine?
>
> Last one, the type names. Fluss spells types two ways: INT and
> TIMESTAMP_LTZ
> in SQL, INTEGER and TIMESTAMP_WITH_LOCAL_TIME_ZONE in DataTypeJsonSerde.
> The
> FIP uses the SQL names with the serde's JSON shape, so I don't think the
> examples parse as written.
>
> Which did you intend? If it's the SQL names, the gateway has to convert
> them
> before calling CreateTable, and that mapping is worth writing into the FIP.
>
> Thank you!
>
> -- Anton
>
> [1] https://github.com/apache/fluss/pull/3463
>
> пн, 3 авг. 2026 г. в 16:02, David Reger <da...@gmail.com>:
> >
> > I think this also relates to one of the architectural discussions we've
> > recently had around FIP-32 [1].
> >
> > My understanding is that there is a useful distinction between *native
> > Fluss operations* and *SQL/query execution*.
> >
> > For example, writes, metadata operations, and potentially lightweight
> > native reads (such as point lookups) could continue to build directly on
> > the fluss-rust foundation, whereas DataFusion would primarily serve the
> > SQL/query path. This would allow the REST gateway to remain lightweight
> > while keeping the query layer reusable for SQL-based protocols in the
> > future.
> >
> > I don't think this necessarily changes the scope of FIP-49, but I do
> think
> > it's an important architectural distinction as we evolve the gateway
> beyond
> > the initial REST write MVP.
> >
> > [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
> >
> > Best regards,
> >
> > David
> >
> > Am Fr., 31. Juli 2026 um 05:02 Uhr schrieb Liting Liu (litiliu) via dev <
> > dev@fluss.apache.org>:
> >
> > > Hi， Junbo
> > >
> > >
> > > Thanks for the detailed proposal. I have a couple of questions about
> the
> > > technology choice and the future SQL path.
> > >
> > > For the current scope—REST writes and metadata/DDL—have we considered
> > > implementing the Gateway with a Java/Netty REST stack backed by the
> > > existing Fluss Java client? This would be similar in spirit to how
> Flink
> > > SQL Gateway reuses Flink’s native Java stack and Netty-based REST
> > > infrastructure. The Fluss Java client already supports append, upsert,
> > > delete, partial update, metadata operations, and lookup. It would be
> > > helpful if the FIP could compare this alternative with the proposed
> > >
> > > I also have a question about the future DataFusion integration. Is
> > > DataFusion expected to execute SQL queries inside the same Gateway
> process?
> > > If so, complex or resource-intensive queries could compete with REST
> > > ingestion and metadata operations for CPU and memory, and a query
> failure
> > > or OOM could potentially affect the entire Gateway. Do we plan to
> provide
> > > admission control, memory and concurrency limits, query
> > > timeout/cancellation, spilling, or process-level isolation?
> > >
> > > If the initial read use case is mainly primary-key or prefix lookup,
> could
> > > we expose a lightweight lookup REST API backed directly by the Fluss
> > > client, without introducing DataFusion? DataFusion seems more valuable
> when
> > > the intended scope includes general SQL capabilities such as scans,
> > > filters, aggregations, and joins? If that is the long-term goal, would
> it
> > > be preferable to keep complex SQL execution in a separate optional SQL
> > > service or worker process, so that it does not increase the failure
> domain
> > > of the REST write Gateway?
> > >
> > >
> > > From: Michael Koepf <mi...@outlook.com>
> > > Date: Thursday, July 30, 2026 at 22:15
> > > To: dev@fluss.apache.org <de...@fluss.apache.org>
> > > Subject: Re: [DISCUSS] FIP-49: Fluss Gateway REST API
> > >
> > > Hi Juno,
> > >
> > > Thanks for the FIP, great initiative.
> > >
> > > A question that comes to my mind: Why not expose metrics via vendor
> > > neutral OTEL instead of limiting it to Prometheus?
> > > --
> > > Best,
> > > MKO
> > >
> > >
> > > > On 30.07.2026, at 12:23, Forward Xu <fo...@gmail.com> wrote:
> > > >
> > > > Thanks junbo,
> > > >
> > > > The proposal looks good to me in general. +1
> > > >
> > > > Best,
> > > > Foward
> > > >
> > > > Junbo Wang <be...@gmail.com> 于2026年7月30日周四 11:49写道：
> > > >
> > > >> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST
> > > API.
> > > >>
> > > >> The proposal is available here:
> > > >>
> > >
> https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API
> > > >>
> > > >> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway)
> [1] and
> > > >> does not conflict with it. Following the earlier suggestion to
> split the
> > > >> gateway work into focused FIPs, FIP-49 defines the REST write path
> and
> > > DDL
> > > >> / metadata management in detail, while REST read and the SQL path
> > > remain in
> > > >> the scope of FIP-32.
> > > >>
> > > >> In short, it proposes a standalone, stateless fluss-gateway process
> > > (Rust,
> > > >> axum + tokio, built on fluss-rust) providing:
> > > >>
> > > >> HTTP/JSON ingestion into KV and Log tables, at-least-once, with
> per-row
> > > >> success/failure reporting
> > > >> Full database / table / partition DDL
> > > >> Pluggable client authentication, and a service / user (act-as)
> identity
> > > >> mode towards Fluss
> > > >> Multi-cluster routing, Prometheus metrics, a unified error model
> > > >>
> > > >> No Fluss server-side changes are required— the gateway accesses
> existing
> > > >> clusters purely as a fluss-rust client.
> > > >>
> > > >> Looking forward to your feedback and suggestions.
> > > >>
> > > >>
> > > >>
> > > >> [1]
> https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
> > > >>
> > > >>
> > > >> Best regards,
> > > >> Junbo Wang
> > > >>
> > > >>
> > >
>
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Anton Borisov <bo...@gmail.com>.
Hi Junbo,
Nice proposal, thank you for driving it!
I like the direction overall, +1
My questions below:
1. Have you seen the new cooperative backpressure?
#3463 [1] went in on 31 July, a day after this thread started. Tablet
servers now
send a per-bucket pressure value on PutKv responses, the client throttles on
that by up to 3 seconds by default, and the server can reject with a retriable
StorageBackpressureException when RocksDB is close to stalling.
Inside an HTTP request that becomes invisible latency, the caller waits and
isn't told why. It also looks like the thing your rate limiting wants to be
driven by. A 429 with Retry-After says "slow down" in a way HTTP clients
already understand, and a fixed requests/sec limit can't track what the
cluster is actually doing. Worth pulling into the FIP?
2. What is a "connection" in service mode, and how many are there?
The page says service mode uses "one shared pool per cluster", and
connection.max is a "total connection cap (exceeded -> 429)". But it doesn't
say what a connection is, or how many service mode opens.
It matters because a Fluss Connection isn't a socket. It's documented as
heavy-weight to create, thread-safe and meant to be shared, and each one
carries a 64 MB write buffer, a sender task, a writer id and sockets to every
tablet server. If service mode really is one shared connection, then the 512
cap and its 429 never fire, and it'd be clearer to say so. If it's many,
they'll batch worse than one would, since each accumulator sees only a
fraction of the traffic. Kafka REST Proxy shares a single producer across
requests for that reason, and only goes per-principal when identities differ.
3. Is there a request timeout on writes?
The client retries forever by default, and "leader not available" is one of
the errors it retries. So the row in your example wouldn't fail, it would
wait. And if the gateway stops waiting and reports a failure, does the row
stay queued and land later? That would make "failed" sometimes mean
"succeeded, just later".
4. Which tables get per-row results?
For Log tables without a bucket key you use append_arrow_batch, which returns
one result for the whole batch and stops at the first failure. I don't think
per-row reporting is possible on that path. Is it meant to be all-or-nothing
there?
5. What stops a duplicate when a client retries?
Our idempotence catches the writer resending its own batch. A retried HTTP
request is a different thing - the rows arrive fresh, so nothing links them to
the first attempt. The writer id also belongs to one process, so a second
gateway can't pick up where the first left off.
6. Is per-user authorization on the roadmap?
The page says Fluss-side ACLs see only the gateway account, so every request
reaches the cluster as gateway_svc and the server can't tell one caller from
another. Is something planned for later: which principals may write to which
tables, or is the gateway meant only for deployments where one shared
identity is fine?
Last one, the type names. Fluss spells types two ways: INT and TIMESTAMP_LTZ
in SQL, INTEGER and TIMESTAMP_WITH_LOCAL_TIME_ZONE in DataTypeJsonSerde. The
FIP uses the SQL names with the serde's JSON shape, so I don't think the
examples parse as written.
Which did you intend? If it's the SQL names, the gateway has to convert them
before calling CreateTable, and that mapping is worth writing into the FIP.
Thank you!
-- Anton
[1] https://github.com/apache/fluss/pull/3463
пн, 3 авг. 2026 г. в 16:02, David Reger <da...@gmail.com>:
>
> I think this also relates to one of the architectural discussions we've
> recently had around FIP-32 [1].
>
> My understanding is that there is a useful distinction between *native
> Fluss operations* and *SQL/query execution*.
>
> For example, writes, metadata operations, and potentially lightweight
> native reads (such as point lookups) could continue to build directly on
> the fluss-rust foundation, whereas DataFusion would primarily serve the
> SQL/query path. This would allow the REST gateway to remain lightweight
> while keeping the query layer reusable for SQL-based protocols in the
> future.
>
> I don't think this necessarily changes the scope of FIP-49, but I do think
> it's an important architectural distinction as we evolve the gateway beyond
> the initial REST write MVP.
>
> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
>
> Best regards,
>
> David
>
> Am Fr., 31. Juli 2026 um 05:02 Uhr schrieb Liting Liu (litiliu) via dev <
> dev@fluss.apache.org>:
>
> > Hi， Junbo
> >
> >
> > Thanks for the detailed proposal. I have a couple of questions about the
> > technology choice and the future SQL path.
> >
> > For the current scope—REST writes and metadata/DDL—have we considered
> > implementing the Gateway with a Java/Netty REST stack backed by the
> > existing Fluss Java client? This would be similar in spirit to how Flink
> > SQL Gateway reuses Flink’s native Java stack and Netty-based REST
> > infrastructure. The Fluss Java client already supports append, upsert,
> > delete, partial update, metadata operations, and lookup. It would be
> > helpful if the FIP could compare this alternative with the proposed
> >
> > I also have a question about the future DataFusion integration. Is
> > DataFusion expected to execute SQL queries inside the same Gateway process?
> > If so, complex or resource-intensive queries could compete with REST
> > ingestion and metadata operations for CPU and memory, and a query failure
> > or OOM could potentially affect the entire Gateway. Do we plan to provide
> > admission control, memory and concurrency limits, query
> > timeout/cancellation, spilling, or process-level isolation?
> >
> > If the initial read use case is mainly primary-key or prefix lookup, could
> > we expose a lightweight lookup REST API backed directly by the Fluss
> > client, without introducing DataFusion? DataFusion seems more valuable when
> > the intended scope includes general SQL capabilities such as scans,
> > filters, aggregations, and joins? If that is the long-term goal, would it
> > be preferable to keep complex SQL execution in a separate optional SQL
> > service or worker process, so that it does not increase the failure domain
> > of the REST write Gateway?
> >
> >
> > From: Michael Koepf <mi...@outlook.com>
> > Date: Thursday, July 30, 2026 at 22:15
> > To: dev@fluss.apache.org <de...@fluss.apache.org>
> > Subject: Re: [DISCUSS] FIP-49: Fluss Gateway REST API
> >
> > Hi Juno,
> >
> > Thanks for the FIP, great initiative.
> >
> > A question that comes to my mind: Why not expose metrics via vendor
> > neutral OTEL instead of limiting it to Prometheus?
> > --
> > Best,
> > MKO
> >
> >
> > > On 30.07.2026, at 12:23, Forward Xu <fo...@gmail.com> wrote:
> > >
> > > Thanks junbo,
> > >
> > > The proposal looks good to me in general. +1
> > >
> > > Best,
> > > Foward
> > >
> > > Junbo Wang <be...@gmail.com> 于2026年7月30日周四 11:49写道：
> > >
> > >> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST
> > API.
> > >>
> > >> The proposal is available here:
> > >>
> > https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API
> > >>
> > >> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] and
> > >> does not conflict with it. Following the earlier suggestion to split the
> > >> gateway work into focused FIPs, FIP-49 defines the REST write path and
> > DDL
> > >> / metadata management in detail, while REST read and the SQL path
> > remain in
> > >> the scope of FIP-32.
> > >>
> > >> In short, it proposes a standalone, stateless fluss-gateway process
> > (Rust,
> > >> axum + tokio, built on fluss-rust) providing:
> > >>
> > >> HTTP/JSON ingestion into KV and Log tables, at-least-once, with per-row
> > >> success/failure reporting
> > >> Full database / table / partition DDL
> > >> Pluggable client authentication, and a service / user (act-as) identity
> > >> mode towards Fluss
> > >> Multi-cluster routing, Prometheus metrics, a unified error model
> > >>
> > >> No Fluss server-side changes are required— the gateway accesses existing
> > >> clusters purely as a fluss-rust client.
> > >>
> > >> Looking forward to your feedback and suggestions.
> > >>
> > >>
> > >>
> > >> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
> > >>
> > >>
> > >> Best regards,
> > >> Junbo Wang
> > >>
> > >>
> >
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by David Reger <da...@gmail.com>.
I think this also relates to one of the architectural discussions we've
recently had around FIP-32 [1].
My understanding is that there is a useful distinction between *native
Fluss operations* and *SQL/query execution*.
For example, writes, metadata operations, and potentially lightweight
native reads (such as point lookups) could continue to build directly on
the fluss-rust foundation, whereas DataFusion would primarily serve the
SQL/query path. This would allow the REST gateway to remain lightweight
while keeping the query layer reusable for SQL-based protocols in the
future.
I don't think this necessarily changes the scope of FIP-49, but I do think
it's an important architectural distinction as we evolve the gateway beyond
the initial REST write MVP.
[1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
Best regards,
David
Am Fr., 31. Juli 2026 um 05:02 Uhr schrieb Liting Liu (litiliu) via dev <
dev@fluss.apache.org>:
> Hi， Junbo
>
>
> Thanks for the detailed proposal. I have a couple of questions about the
> technology choice and the future SQL path.
>
> For the current scope—REST writes and metadata/DDL—have we considered
> implementing the Gateway with a Java/Netty REST stack backed by the
> existing Fluss Java client? This would be similar in spirit to how Flink
> SQL Gateway reuses Flink’s native Java stack and Netty-based REST
> infrastructure. The Fluss Java client already supports append, upsert,
> delete, partial update, metadata operations, and lookup. It would be
> helpful if the FIP could compare this alternative with the proposed
>
> I also have a question about the future DataFusion integration. Is
> DataFusion expected to execute SQL queries inside the same Gateway process?
> If so, complex or resource-intensive queries could compete with REST
> ingestion and metadata operations for CPU and memory, and a query failure
> or OOM could potentially affect the entire Gateway. Do we plan to provide
> admission control, memory and concurrency limits, query
> timeout/cancellation, spilling, or process-level isolation?
>
> If the initial read use case is mainly primary-key or prefix lookup, could
> we expose a lightweight lookup REST API backed directly by the Fluss
> client, without introducing DataFusion? DataFusion seems more valuable when
> the intended scope includes general SQL capabilities such as scans,
> filters, aggregations, and joins? If that is the long-term goal, would it
> be preferable to keep complex SQL execution in a separate optional SQL
> service or worker process, so that it does not increase the failure domain
> of the REST write Gateway?
>
>
> From: Michael Koepf <mi...@outlook.com>
> Date: Thursday, July 30, 2026 at 22:15
> To: dev@fluss.apache.org <de...@fluss.apache.org>
> Subject: Re: [DISCUSS] FIP-49: Fluss Gateway REST API
>
> Hi Juno,
>
> Thanks for the FIP, great initiative.
>
> A question that comes to my mind: Why not expose metrics via vendor
> neutral OTEL instead of limiting it to Prometheus?
> --
> Best,
> MKO
>
>
> > On 30.07.2026, at 12:23, Forward Xu <fo...@gmail.com> wrote:
> >
> > Thanks junbo,
> >
> > The proposal looks good to me in general. +1
> >
> > Best,
> > Foward
> >
> > Junbo Wang <be...@gmail.com> 于2026年7月30日周四 11:49写道：
> >
> >> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST
> API.
> >>
> >> The proposal is available here:
> >>
> https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API
> >>
> >> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] and
> >> does not conflict with it. Following the earlier suggestion to split the
> >> gateway work into focused FIPs, FIP-49 defines the REST write path and
> DDL
> >> / metadata management in detail, while REST read and the SQL path
> remain in
> >> the scope of FIP-32.
> >>
> >> In short, it proposes a standalone, stateless fluss-gateway process
> (Rust,
> >> axum + tokio, built on fluss-rust) providing:
> >>
> >> HTTP/JSON ingestion into KV and Log tables, at-least-once, with per-row
> >> success/failure reporting
> >> Full database / table / partition DDL
> >> Pluggable client authentication, and a service / user (act-as) identity
> >> mode towards Fluss
> >> Multi-cluster routing, Prometheus metrics, a unified error model
> >>
> >> No Fluss server-side changes are required— the gateway accesses existing
> >> clusters purely as a fluss-rust client.
> >>
> >> Looking forward to your feedback and suggestions.
> >>
> >>
> >>
> >> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
> >>
> >>
> >> Best regards,
> >> Junbo Wang
> >>
> >>
>
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by "Liting Liu (litiliu) via dev" <de...@fluss.apache.org>.
Hi， Junbo
Thanks for the detailed proposal. I have a couple of questions about the technology choice and the future SQL path.
For the current scope—REST writes and metadata/DDL—have we considered implementing the Gateway with a Java/Netty REST stack backed by the existing Fluss Java client? This would be similar in spirit to how Flink SQL Gateway reuses Flink’s native Java stack and Netty-based REST infrastructure. The Fluss Java client already supports append, upsert, delete, partial update, metadata operations, and lookup. It would be helpful if the FIP could compare this alternative with the proposed
I also have a question about the future DataFusion integration. Is DataFusion expected to execute SQL queries inside the same Gateway process? If so, complex or resource-intensive queries could compete with REST ingestion and metadata operations for CPU and memory, and a query failure or OOM could potentially affect the entire Gateway. Do we plan to provide admission control, memory and concurrency limits, query timeout/cancellation, spilling, or process-level isolation?
If the initial read use case is mainly primary-key or prefix lookup, could we expose a lightweight lookup REST API backed directly by the Fluss client, without introducing DataFusion? DataFusion seems more valuable when the intended scope includes general SQL capabilities such as scans, filters, aggregations, and joins? If that is the long-term goal, would it be preferable to keep complex SQL execution in a separate optional SQL service or worker process, so that it does not increase the failure domain of the REST write Gateway?
From: Michael Koepf <mi...@outlook.com>
Date: Thursday, July 30, 2026 at 22:15
To: dev@fluss.apache.org <de...@fluss.apache.org>
Subject: Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Hi Juno,
Thanks for the FIP, great initiative.
A question that comes to my mind: Why not expose metrics via vendor neutral OTEL instead of limiting it to Prometheus?
--
Best,
MKO
> On 30.07.2026, at 12:23, Forward Xu <fo...@gmail.com> wrote:
>
> Thanks junbo,
>
> The proposal looks good to me in general. +1
>
> Best,
> Foward
>
> Junbo Wang <be...@gmail.com> 于2026年7月30日周四 11:49写道：
>
>> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST API.
>>
>> The proposal is available here:
>> https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API
>>
>> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] and
>> does not conflict with it. Following the earlier suggestion to split the
>> gateway work into focused FIPs, FIP-49 defines the REST write path and DDL
>> / metadata management in detail, while REST read and the SQL path remain in
>> the scope of FIP-32.
>>
>> In short, it proposes a standalone, stateless fluss-gateway process (Rust,
>> axum + tokio, built on fluss-rust) providing:
>>
>> HTTP/JSON ingestion into KV and Log tables, at-least-once, with per-row
>> success/failure reporting
>> Full database / table / partition DDL
>> Pluggable client authentication, and a service / user (act-as) identity
>> mode towards Fluss
>> Multi-cluster routing, Prometheus metrics, a unified error model
>>
>> No Fluss server-side changes are required— the gateway accesses existing
>> clusters purely as a fluss-rust client.
>>
>> Looking forward to your feedback and suggestions.
>>
>>
>>
>> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
>>
>>
>> Best regards,
>> Junbo Wang
>>
>>
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Michael Koepf <mi...@outlook.com>.
Hi Juno,
Thanks for the FIP, great initiative.
A question that comes to my mind: Why not expose metrics via vendor neutral OTEL instead of limiting it to Prometheus?
--
Best,
MKO
> On 30.07.2026, at 12:23, Forward Xu <fo...@gmail.com> wrote:
>
> Thanks junbo,
>
> The proposal looks good to me in general. +1
>
> Best,
> Foward
>
> Junbo Wang <be...@gmail.com> 于2026年7月30日周四 11:49写道：
>
>> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST API.
>>
>> The proposal is available here:
>> https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API
>>
>> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] and
>> does not conflict with it. Following the earlier suggestion to split the
>> gateway work into focused FIPs, FIP-49 defines the REST write path and DDL
>> / metadata management in detail, while REST read and the SQL path remain in
>> the scope of FIP-32.
>>
>> In short, it proposes a standalone, stateless fluss-gateway process (Rust,
>> axum + tokio, built on fluss-rust) providing:
>>
>> HTTP/JSON ingestion into KV and Log tables, at-least-once, with per-row
>> success/failure reporting
>> Full database / table / partition DDL
>> Pluggable client authentication, and a service / user (act-as) identity
>> mode towards Fluss
>> Multi-cluster routing, Prometheus metrics, a unified error model
>>
>> No Fluss server-side changes are required— the gateway accesses existing
>> clusters purely as a fluss-rust client.
>>
>> Looking forward to your feedback and suggestions.
>>
>>
>>
>> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
>>
>>
>> Best regards,
>> Junbo Wang
>>
>>
Re: [DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Forward Xu <fo...@gmail.com>.
Thanks junbo,
The proposal looks good to me in general. +1
Best,
Foward
Junbo Wang <be...@gmail.com> 于2026年7月30日周四 11:49写道：
> Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST API.
>
> The proposal is available here:
> https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API
>
> This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] and
> does not conflict with it. Following the earlier suggestion to split the
> gateway work into focused FIPs, FIP-49 defines the REST write path and DDL
> / metadata management in detail, while REST read and the SQL path remain in
> the scope of FIP-32.
>
> In short, it proposes a standalone, stateless fluss-gateway process (Rust,
> axum + tokio, built on fluss-rust) providing:
>
> HTTP/JSON ingestion into KV and Log tables, at-least-once, with per-row
> success/failure reporting
> Full database / table / partition DDL
> Pluggable client authentication, and a service / user (act-as) identity
> mode towards Fluss
> Multi-cluster routing, Prometheus metrics, a unified error model
>
> No Fluss server-side changes are required— the gateway accesses existing
> clusters purely as a fluss-rust client.
>
> Looking forward to your feedback and suggestions.
>
>
>
> [1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
>
>
> Best regards,
> Junbo Wang
>
>
[DISCUSS] FIP-49: Fluss Gateway REST API
Posted by Junbo Wang <be...@gmail.com>.
Hi all, I'd like to start a discussion on FIP-49: Fluss Gateway REST API.
The proposal is available here: https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334561/FIP-49+Fluss+Gateway+REST+API
This FIP is an extension of FIP-32 (Multiprotocol Query Gateway) [1] and does not conflict with it. Following the earlier suggestion to split the gateway work into focused FIPs, FIP-49 defines the REST write path and DDL / metadata management in detail, while REST read and the SQL path remain in the scope of FIP-32.
In short, it proposes a standalone, stateless fluss-gateway process (Rust, axum + tokio, built on fluss-rust) providing:
HTTP/JSON ingestion into KV and Log tables, at-least-once, with per-row success/failure reporting
Full database / table / partition DDL
Pluggable client authentication, and a service / user (act-as) identity mode towards Fluss
Multi-cluster routing, Prometheus metrics, a unified error model
No Fluss server-side changes are required— the gateway accesses existing clusters purely as a fluss-rust client.
Looking forward to your feedback and suggestions.
[1] https://lists.apache.org/thread/sf0qfxhr9j2g0dtxvp586p9zknxvpx6x
Best regards,
Junbo Wang
Re: [DISCUSS] FIP-12: Server Dynamic Config
Posted by Hongshun Wang <lo...@gmail.com>.
Hi everyone,
I've updated the "Future Work" section to include a proposed mechanism for
tracking whether dynamic configurations have been successfully applied
across all servers. I've also added an explanation of why not implement it
in this FIP now.
Thank you all for your valuable feedback. If there are no further concerns,
I plan to start the vote shortly.
Best regards,
Hongshun
On Thu, Aug 7, 2025 at 11:51 AM Hongshun Wang <lo...@gmail.com>
wrote:
> Hi devs,
>
> I'd like to start a discussion about FIP-12: Server Dynamic Config[1].
> Currently, any changes to the server.yaml configuration in a Fluss cluster
> require a full restart of the cluster, which negatively impacts stability
> and availability. To improve operational agility and reduce downtime, we
> propose introducing dynamic configuration capabilities, enabling runtime
> modification of key parameters—such as enabling/disabling lake-streaming
> integration features or managing user accounts—without requiring service
> interruption.
>
> The POC[2] code is provided to enable lake format. You can try and give
> some advice.
>
> Best
> Hongshun
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-12%3A+Server+Dynamic+Config
> [2] https://github.com/loserwang1024/fluss/tree/poc-dymanic-config.
>