You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fluss.apache.org by yuxia <lu...@alumni.sjtu.edu.cn> on 2025/07/03 08:51:03 UTC
[DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Hi,
Fluss currently supports tiering data to Apache Paimon, enabling cost-effective storage management for warm/cold data. However, the lack of native Iceberg tiering support limits flexibility and ecosystem integration for users who rely on Iceberg’s open table format.
To address this gap, I’d like to propose FIP-3: Support Tiering Fluss Data to Iceberg[1] which aims to integrate Iceberg into Fluss’s tiering capabilities.
Welcome your feedback and suggestions on this proposal. Looking forward to a productive discussion!
[1]: https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
Best regards,
Yuxia
Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Thanks Leonard for the feedback. We‘ll have a detail user document to explain it.
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Leonard Xu" <xb...@gmail.com>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期四, 2025年 7 月 10日 下午 10:36:20
主题: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Thanks yuxia for bringing this discussion, the FIP generally looks good to me.
The only feedback is that Fluss’ two kinds table with different bucket key combination need different mapping rules to iceberg table, a table to list the mapping will help developers to understand the relationship better, although I think they’re a little complex for users who lacks the context.
Best,
Leonard
> 2025 7月 3 16:51，yuxia <lu...@alumni.sjtu.edu.cn> 写道：
>
> Hi,
>
> Fluss currently supports tiering data to Apache Paimon, enabling cost-effective storage management for warm/cold data. However, the lack of native Iceberg tiering support limits flexibility and ecosystem integration for users who rely on Iceberg’s open table format.
>
> To address this gap, I’d like to propose FIP-3: Support Tiering Fluss Data to Iceberg[1] which aims to integrate Iceberg into Fluss’s tiering capabilities.
>
> Welcome your feedback and suggestions on this proposal. Looking forward to a productive discussion!
>
> [1]: https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
>
> Best regards,
> Yuxia
>
Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by Leonard Xu <xb...@gmail.com>.
Thanks yuxia for bringing this discussion, the FIP generally looks good to me.
The only feedback is that Fluss’ two kinds table with different bucket key combination need different mapping rules to iceberg table, a table to list the mapping will help developers to understand the relationship better, although I think they’re a little complex for users who lacks the context.
Best,
Leonard
> 2025 7月 3 16:51，yuxia <lu...@alumni.sjtu.edu.cn> 写道：
>
> Hi,
>
> Fluss currently supports tiering data to Apache Paimon, enabling cost-effective storage management for warm/cold data. However, the lack of native Iceberg tiering support limits flexibility and ecosystem integration for users who rely on Iceberg’s open table format.
>
> To address this gap, I’d like to propose FIP-3: Support Tiering Fluss Data to Iceberg[1] which aims to integrate Iceberg into Fluss’s tiering capabilities.
>
> Welcome your feedback and suggestions on this proposal. Looking forward to a productive discussion!
>
> [1]: https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
>
> Best regards,
> Yuxia
>
Re: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by Mehul Batra <me...@gmail.com>.
Thanks, Jark, for the feedback!
I agree with you and Yuxia that for the current version, we can proceed
with a single boolean flag for the maintenance option.
That said, I’ve seen that in S3-backed tables, users often prefer having
separate switches for different maintenance tasks (like compaction and
snapshot management) to control them individually. It’s probably something
we can consider revisiting in the future.
Best Regards,
Mehul Batra
On Thu, Jul 10, 2025 at 11:47 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Thanks for Jark's comments. I agree with Jark that it should allow user to
> disable the automated maintenance while tiering to lake. I introduce
> a new boolean option `table.datalake.auto-maintenance`(true by default) to
> configure whether to do maintenance tasks to per table in lake tiering
> service.
> The FIP has been updated.
>
> If have no any other comments, I'll start the vote process today later.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Jark Wu" <im...@gmail.com>
> 收件人: "dev" <de...@fluss.apache.org>
> 发送时间: 星期四, 2025年 7 月 10日 上午 10:50:55
> 主题: Re: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
>
> Thank you all for the healthy discussion.
>
> My view is that all automated maintenance tasks — such as compactions, data
> expiration, and cleanup — should be supported by the tiering service, but
> remain configurable.
>
> These features can be enabled by default to support simple end-to-end use
> cases, without requiring users to rely on an external service.
>
> However, they should also be optional. In many cases, users may already
> have or prefer to use a dedicated maintenance service for their lakehouse
> tables. Such specialized services are often more capable and can manage
> maintenance tasks across all lake tables, not only the fluss tiered tables.
>
>
> Besides, +1 for the design proposal. I think we can kick off a vote now.
>
> Best,
> Jark
>
> On Tue, 8 Jul 2025 at 15:24, Mehul Batra <me...@gmail.com> wrote:
>
> > Hi Yuxia,
> >
> > Thanks for the clarification.
> >
> > It's good to know that compaction/expiring-snapshots will be addressed in
> > the initial version, and I completely understand your point on the
> > complexity of orphan file cleanup. I agree it's better to avoid
> over-design
> > at this stage and evolve as needed based on future usage patterns and
> > feedback.
> >
> > Also, thanks for confirming that snapshot expiration will be triggered
> > explicitly via the LakeCommitter. That clears things up for me.
> >
> > Looking forward to working on this with you and the community, seeing how
> > this evolves!
> >
> > Best regards,
> > Mehul
> > On Tue, Jul 8, 2025 at 7:16 AM yuxia <lu...@alumni.sjtu.edu.cn>
> wrote:
> >
> > > Hi, Mehul
> > >
> > > "Tableflow automates table maintenance by compacting and cleaning up
> > small
> > > files generated by continuous streaming data in object storage."
> > > Seems table flow supports compacting, which is covered in this FIP.
> > > Haven't seen orphan file cleanup in table flow.
> > > Orphan file cleanup is not straightforward and a little of complex,
> which
> > > required to list all files, and compare with the files in iceberg
> > manifest
> > > to find the orphan files.
> > > I still prefer not to introduce the complexity currently for the first
> > > version of iceberg support, which will cause overdesign. Let's just see
> > in
> > > the future what'll happen.
> > >
> > > As for snapshot expiration. Yes, LakeCommitter should trigger the
> > snapshot
> > > expiration action explicitly. It's a slight operation.
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > ----- 原始邮件 -----
> > > 发件人: "Mehul Batra" <me...@gmail.com>
> > > 收件人: "dev" <de...@fluss.apache.org>
> > > 发送时间: 星期二, 2025年 7 月 08日 上午 4:26:19
> > > 主题: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> > >
> > > Hi Yuxia, Cheng,
> > >
> > > Thank you both for the insights.
> > >
> > > From a user’s perspective, I believe our goal should be to abstract
> away
> > as
> > > much operational complexity as possible. For example, TableFlow handles
> > > both data writing and maintenance seamlessly for the user, which avoids
> > the
> > > burden of running separate processes.
> > >
> > >
> >
> https://docs.confluent.io/cloud/current/topics/tableflow/overview.html#table-maintenance-and-optimizations
> > >
> > > In Fluss Integration, if users are expected to run a separate
> maintenance
> > > job (e.g., for snapshot expiration or orphan file cleanup), there's a
> > real
> > > risk of job overlap and failure, especially due to optimistic
> concurrency
> > > issues when both (tiering & maintenance) try to commit around the same
> > > time.
> > >
> > > Yuxia, you mentioned that the LakeCommitter will respect the
> > > history.expire.max-snapshot-age-ms property (similar to Paimon). I just
> > > wanted to clarify, while the property sets the retention policy, we
> still
> > > need to trigger the snapshot expiration action explicitly. Do we
> envision
> > > Fluss's tiering job playing that role?
> > >
> > > If so, that would be a great win, it could help automate snapshot
> > > expiration and indirectly clean up orphan files, making things much
> > > smoother for users.
> > >
> > > Please correct me if I’ve misunderstood anything.
> > >
> > > Best regards,
> > > Mehul
> > >
> > > On Mon, Jul 7, 2025 at 11:32 AM Wang Cheng <34...@qq.com.invalid>
> > > wrote:
> > >
> > > > Hi Mehul,
> > > >
> > > >
> > > > I agree with Yuxia's point. We should leave such table maintenance
> work
> > > > like expiring snapshots and deleting orphan files to Iceberg
> users
> > > > rather than relying on Fluss tiering job.
> > > >
> > > >
> > > >
> > > > Regards,
> > > > Cheng
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ------------------ Original ------------------
> > > > From:
> > > > "dev"
> > > > <
> > > > mehulbatra1997@gmail.com>;
> > > > Date: Sat, Jul 5, 2025 11:38 PM
> > > > To: "dev"<dev@fluss.apache.org>;
> > > >
> > > > Subject: Re: [DISCUSS] FIP-3: Support tiering Fluss data to
> > Iceberg
> > > >
> > > >
> > > >
> > > > Hi Yuxia,
> > > > Great, that sounds good to me and will help the user to have a better
> > > read
> > > > latency.
> > > > How about the Snapshot expiration (to regulate metadata) and removing
> > the
> > > > orphan files(which are no longer referenced or dangling files of
> failed
> > > > tasks)?
> > > > Are we planning to introduce them as part of automated maintenance
> > > provided
> > > > by the Fluss cluster?
> > > > Warm regards,
> > > > Mehul Batra
> > > >
> > > > On Fri, Jul 4, 2025 at 5:02 PM yuxia <luoyuxia@alumni.sjtu.edu.cn
> >
> > > > wrote:
> > > >
> > > > > Hi, Mehul.
> > > > > Thanks for your attention. I think we don't need to introduce an
> > > extra
> > > > > post-commit hook to manage small files. In the design, all files
> > > that
> > > > belong
> > > > > to same bucket(in iceberg, it'll be same partition) be
> distributed
> > > to
> > > > same
> > > > > task to write. So, the task can compact these small files then
> for
> > > the
> > > > > partition.
> > > > > As this FIP said, while creating IcebergLakeWriter in one round
> of
> > > > > tiering, the writer can scan manifest to know the files in this
> > > > bucket, if
> > > > > found compaction is available, it can
> > > > > compact these files while writing new files. We have a similar
> > logic
> > > > for
> > > > > tiering to paimon.
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > > ----- 原始邮件 -----
> > > > > 发件人: "Mehul Batra" <mehulbatra1997@gmail.com>
> > > > > 收件人: "dev" <dev@fluss.apache.org>
> > > > > 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> > > > > 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> > > > >
> > > > > +1 This will help us to address the missing table format and
> > provide
> > > > better
> > > > > ecosystem interoperability. Iceberg's growing adoption in the
> data
> > > > > lakehouse space makes this a valuable addition to Fluss's
> tiering
> > > > > capabilities.
> > > > > Are there any plans to integrate the Maintenance services as
> part
> > of
> > > > > tiering itself as a post-commit hook to manage small files?
> > > > > Warm regards,
> > > > > Mehul Batra
> > > > >
> > > > > On Thu, Jul 3, 2025 at 2:24 PM yuxia <
> luoyuxia@alumni.sjtu.edu.cn
> > > >
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Fluss currently supports tiering data to Apache Paimon,
> > > enabling
> > > > > > cost-effective storage management for warm/cold data.
> > However,
> > > > the lack
> > > > > of
> > > > > > native Iceberg tiering support limits flexibility and
> > ecosystem
> > > > > integration
> > > > > > for users who rely on Iceberg’s open table format.
> > > > > >
> > > > > > To address this gap, I’d like to propose FIP-3: Support
> > Tiering
> > > > Fluss
> > > > > Data
> > > > > > to Iceberg[1] which aims to integrate Iceberg into Fluss’s
> > > > tiering
> > > > > > capabilities.
> > > > > >
> > > > > > Welcome your feedback and suggestions on this proposal.
> > Looking
> > > > forward
> > > > > to
> > > > > > a productive discussion!
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> > > > >
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg>
> > > >;
> > > > >
> > > > > > Best regards,
> > > > > > Yuxia
> > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Thanks for Jark's comments. I agree with Jark that it should allow user to disable the automated maintenance while tiering to lake. I introduce
a new boolean option `table.datalake.auto-maintenance`(true by default) to configure whether to do maintenance tasks to per table in lake tiering service.
The FIP has been updated.
If have no any other comments, I'll start the vote process today later.
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Jark Wu" <im...@gmail.com>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期四, 2025年 7 月 10日 上午 10:50:55
主题: Re: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Thank you all for the healthy discussion.
My view is that all automated maintenance tasks — such as compactions, data
expiration, and cleanup — should be supported by the tiering service, but
remain configurable.
These features can be enabled by default to support simple end-to-end use
cases, without requiring users to rely on an external service.
However, they should also be optional. In many cases, users may already
have or prefer to use a dedicated maintenance service for their lakehouse
tables. Such specialized services are often more capable and can manage
maintenance tasks across all lake tables, not only the fluss tiered tables.
Besides, +1 for the design proposal. I think we can kick off a vote now.
Best,
Jark
On Tue, 8 Jul 2025 at 15:24, Mehul Batra <me...@gmail.com> wrote:
> Hi Yuxia,
>
> Thanks for the clarification.
>
> It's good to know that compaction/expiring-snapshots will be addressed in
> the initial version, and I completely understand your point on the
> complexity of orphan file cleanup. I agree it's better to avoid over-design
> at this stage and evolve as needed based on future usage patterns and
> feedback.
>
> Also, thanks for confirming that snapshot expiration will be triggered
> explicitly via the LakeCommitter. That clears things up for me.
>
> Looking forward to working on this with you and the community, seeing how
> this evolves!
>
> Best regards,
> Mehul
> On Tue, Jul 8, 2025 at 7:16 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> > Hi, Mehul
> >
> > "Tableflow automates table maintenance by compacting and cleaning up
> small
> > files generated by continuous streaming data in object storage."
> > Seems table flow supports compacting, which is covered in this FIP.
> > Haven't seen orphan file cleanup in table flow.
> > Orphan file cleanup is not straightforward and a little of complex, which
> > required to list all files, and compare with the files in iceberg
> manifest
> > to find the orphan files.
> > I still prefer not to introduce the complexity currently for the first
> > version of iceberg support, which will cause overdesign. Let's just see
> in
> > the future what'll happen.
> >
> > As for snapshot expiration. Yes, LakeCommitter should trigger the
> snapshot
> > expiration action explicitly. It's a slight operation.
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Mehul Batra" <me...@gmail.com>
> > 收件人: "dev" <de...@fluss.apache.org>
> > 发送时间: 星期二, 2025年 7 月 08日 上午 4:26:19
> > 主题: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> >
> > Hi Yuxia, Cheng,
> >
> > Thank you both for the insights.
> >
> > From a user’s perspective, I believe our goal should be to abstract away
> as
> > much operational complexity as possible. For example, TableFlow handles
> > both data writing and maintenance seamlessly for the user, which avoids
> the
> > burden of running separate processes.
> >
> >
> https://docs.confluent.io/cloud/current/topics/tableflow/overview.html#table-maintenance-and-optimizations
> >
> > In Fluss Integration, if users are expected to run a separate maintenance
> > job (e.g., for snapshot expiration or orphan file cleanup), there's a
> real
> > risk of job overlap and failure, especially due to optimistic concurrency
> > issues when both (tiering & maintenance) try to commit around the same
> > time.
> >
> > Yuxia, you mentioned that the LakeCommitter will respect the
> > history.expire.max-snapshot-age-ms property (similar to Paimon). I just
> > wanted to clarify, while the property sets the retention policy, we still
> > need to trigger the snapshot expiration action explicitly. Do we envision
> > Fluss's tiering job playing that role?
> >
> > If so, that would be a great win, it could help automate snapshot
> > expiration and indirectly clean up orphan files, making things much
> > smoother for users.
> >
> > Please correct me if I’ve misunderstood anything.
> >
> > Best regards,
> > Mehul
> >
> > On Mon, Jul 7, 2025 at 11:32 AM Wang Cheng <34...@qq.com.invalid>
> > wrote:
> >
> > > Hi Mehul,
> > >
> > >
> > > I agree with Yuxia's point. We should leave such table maintenance work
> > > like expiring snapshots and deleting orphan files to Iceberg users
> > > rather than relying on Fluss tiering job.
> > >
> > >
> > >
> > > Regards,
> > > Cheng
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ------------------ Original ------------------
> > > From:
> > > "dev"
> > > <
> > > mehulbatra1997@gmail.com>;
> > > Date: Sat, Jul 5, 2025 11:38 PM
> > > To: "dev"<dev@fluss.apache.org>;
> > >
> > > Subject: Re: [DISCUSS] FIP-3: Support tiering Fluss data to
> Iceberg
> > >
> > >
> > >
> > > Hi Yuxia,
> > > Great, that sounds good to me and will help the user to have a better
> > read
> > > latency.
> > > How about the Snapshot expiration (to regulate metadata) and removing
> the
> > > orphan files(which are no longer referenced or dangling files of failed
> > > tasks)?
> > > Are we planning to introduce them as part of automated maintenance
> > provided
> > > by the Fluss cluster?
> > > Warm regards,
> > > Mehul Batra
> > >
> > > On Fri, Jul 4, 2025 at 5:02 PM yuxia <luoyuxia@alumni.sjtu.edu.cn>
> > > wrote:
> > >
> > > > Hi, Mehul.
> > > > Thanks for your attention. I think we don't need to introduce an
> > extra
> > > > post-commit hook to manage small files. In the design, all files
> > that
> > > belong
> > > > to same bucket(in iceberg, it'll be same partition) be distributed
> > to
> > > same
> > > > task to write. So, the task can compact these small files then for
> > the
> > > > partition.
> > > > As this FIP said, while creating IcebergLakeWriter in one round of
> > > > tiering, the writer can scan manifest to know the files in this
> > > bucket, if
> > > > found compaction is available, it can
> > > > compact these files while writing new files. We have a similar
> logic
> > > for
> > > > tiering to paimon.
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > ----- 原始邮件 -----
> > > > 发件人: "Mehul Batra" <mehulbatra1997@gmail.com>
> > > > 收件人: "dev" <dev@fluss.apache.org>
> > > > 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> > > > 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> > > >
> > > > +1 This will help us to address the missing table format and
> provide
> > > better
> > > > ecosystem interoperability. Iceberg's growing adoption in the data
> > > > lakehouse space makes this a valuable addition to Fluss's tiering
> > > > capabilities.
> > > > Are there any plans to integrate the Maintenance services as part
> of
> > > > tiering itself as a post-commit hook to manage small files?
> > > > Warm regards,
> > > > Mehul Batra
> > > >
> > > > On Thu, Jul 3, 2025 at 2:24 PM yuxia <luoyuxia@alumni.sjtu.edu.cn
> > >
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Fluss currently supports tiering data to Apache Paimon,
> > enabling
> > > > > cost-effective storage management for warm/cold data.
> However,
> > > the lack
> > > > of
> > > > > native Iceberg tiering support limits flexibility and
> ecosystem
> > > > integration
> > > > > for users who rely on Iceberg’s open table format.
> > > > >
> > > > > To address this gap, I’d like to propose FIP-3: Support
> Tiering
> > > Fluss
> > > > Data
> > > > > to Iceberg[1] which aims to integrate Iceberg into Fluss’s
> > > tiering
> > > > > capabilities.
> > > > >
> > > > > Welcome your feedback and suggestions on this proposal.
> Looking
> > > forward
> > > > to
> > > > > a productive discussion!
> > > > >
> > > > > [1]:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> > > >
> > > <
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg>
> > >;
> > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > >
> > > >
> >
>
Re: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by Jark Wu <im...@gmail.com>.
Thank you all for the healthy discussion.
My view is that all automated maintenance tasks — such as compactions, data
expiration, and cleanup — should be supported by the tiering service, but
remain configurable.
These features can be enabled by default to support simple end-to-end use
cases, without requiring users to rely on an external service.
However, they should also be optional. In many cases, users may already
have or prefer to use a dedicated maintenance service for their lakehouse
tables. Such specialized services are often more capable and can manage
maintenance tasks across all lake tables, not only the fluss tiered tables.
Besides, +1 for the design proposal. I think we can kick off a vote now.
Best,
Jark
On Tue, 8 Jul 2025 at 15:24, Mehul Batra <me...@gmail.com> wrote:
> Hi Yuxia,
>
> Thanks for the clarification.
>
> It's good to know that compaction/expiring-snapshots will be addressed in
> the initial version, and I completely understand your point on the
> complexity of orphan file cleanup. I agree it's better to avoid over-design
> at this stage and evolve as needed based on future usage patterns and
> feedback.
>
> Also, thanks for confirming that snapshot expiration will be triggered
> explicitly via the LakeCommitter. That clears things up for me.
>
> Looking forward to working on this with you and the community, seeing how
> this evolves!
>
> Best regards,
> Mehul
> On Tue, Jul 8, 2025 at 7:16 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> > Hi, Mehul
> >
> > "Tableflow automates table maintenance by compacting and cleaning up
> small
> > files generated by continuous streaming data in object storage."
> > Seems table flow supports compacting, which is covered in this FIP.
> > Haven't seen orphan file cleanup in table flow.
> > Orphan file cleanup is not straightforward and a little of complex, which
> > required to list all files, and compare with the files in iceberg
> manifest
> > to find the orphan files.
> > I still prefer not to introduce the complexity currently for the first
> > version of iceberg support, which will cause overdesign. Let's just see
> in
> > the future what'll happen.
> >
> > As for snapshot expiration. Yes, LakeCommitter should trigger the
> snapshot
> > expiration action explicitly. It's a slight operation.
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Mehul Batra" <me...@gmail.com>
> > 收件人: "dev" <de...@fluss.apache.org>
> > 发送时间: 星期二, 2025年 7 月 08日 上午 4:26:19
> > 主题: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> >
> > Hi Yuxia, Cheng,
> >
> > Thank you both for the insights.
> >
> > From a user’s perspective, I believe our goal should be to abstract away
> as
> > much operational complexity as possible. For example, TableFlow handles
> > both data writing and maintenance seamlessly for the user, which avoids
> the
> > burden of running separate processes.
> >
> >
> https://docs.confluent.io/cloud/current/topics/tableflow/overview.html#table-maintenance-and-optimizations
> >
> > In Fluss Integration, if users are expected to run a separate maintenance
> > job (e.g., for snapshot expiration or orphan file cleanup), there's a
> real
> > risk of job overlap and failure, especially due to optimistic concurrency
> > issues when both (tiering & maintenance) try to commit around the same
> > time.
> >
> > Yuxia, you mentioned that the LakeCommitter will respect the
> > history.expire.max-snapshot-age-ms property (similar to Paimon). I just
> > wanted to clarify, while the property sets the retention policy, we still
> > need to trigger the snapshot expiration action explicitly. Do we envision
> > Fluss's tiering job playing that role?
> >
> > If so, that would be a great win, it could help automate snapshot
> > expiration and indirectly clean up orphan files, making things much
> > smoother for users.
> >
> > Please correct me if I’ve misunderstood anything.
> >
> > Best regards,
> > Mehul
> >
> > On Mon, Jul 7, 2025 at 11:32 AM Wang Cheng <34...@qq.com.invalid>
> > wrote:
> >
> > > Hi Mehul,
> > >
> > >
> > > I agree with Yuxia's point. We should leave such table maintenance work
> > > like expiring snapshots and deleting orphan files to Iceberg users
> > > rather than relying on Fluss tiering job.
> > >
> > >
> > >
> > > Regards,
> > > Cheng
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ------------------ Original ------------------
> > > From:
> > > "dev"
> > > <
> > > mehulbatra1997@gmail.com>;
> > > Date: Sat, Jul 5, 2025 11:38 PM
> > > To: "dev"<dev@fluss.apache.org>;
> > >
> > > Subject: Re: [DISCUSS] FIP-3: Support tiering Fluss data to
> Iceberg
> > >
> > >
> > >
> > > Hi Yuxia,
> > > Great, that sounds good to me and will help the user to have a better
> > read
> > > latency.
> > > How about the Snapshot expiration (to regulate metadata) and removing
> the
> > > orphan files(which are no longer referenced or dangling files of failed
> > > tasks)?
> > > Are we planning to introduce them as part of automated maintenance
> > provided
> > > by the Fluss cluster?
> > > Warm regards,
> > > Mehul Batra
> > >
> > > On Fri, Jul 4, 2025 at 5:02 PM yuxia <luoyuxia@alumni.sjtu.edu.cn>
> > > wrote:
> > >
> > > > Hi, Mehul.
> > > > Thanks for your attention. I think we don't need to introduce an
> > extra
> > > > post-commit hook to manage small files. In the design, all files
> > that
> > > belong
> > > > to same bucket(in iceberg, it'll be same partition) be distributed
> > to
> > > same
> > > > task to write. So, the task can compact these small files then for
> > the
> > > > partition.
> > > > As this FIP said, while creating IcebergLakeWriter in one round of
> > > > tiering, the writer can scan manifest to know the files in this
> > > bucket, if
> > > > found compaction is available, it can
> > > > compact these files while writing new files. We have a similar
> logic
> > > for
> > > > tiering to paimon.
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > ----- 原始邮件 -----
> > > > 发件人: "Mehul Batra" <mehulbatra1997@gmail.com>
> > > > 收件人: "dev" <dev@fluss.apache.org>
> > > > 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> > > > 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> > > >
> > > > +1 This will help us to address the missing table format and
> provide
> > > better
> > > > ecosystem interoperability. Iceberg's growing adoption in the data
> > > > lakehouse space makes this a valuable addition to Fluss's tiering
> > > > capabilities.
> > > > Are there any plans to integrate the Maintenance services as part
> of
> > > > tiering itself as a post-commit hook to manage small files?
> > > > Warm regards,
> > > > Mehul Batra
> > > >
> > > > On Thu, Jul 3, 2025 at 2:24 PM yuxia <luoyuxia@alumni.sjtu.edu.cn
> > >
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Fluss currently supports tiering data to Apache Paimon,
> > enabling
> > > > > cost-effective storage management for warm/cold data.
> However,
> > > the lack
> > > > of
> > > > > native Iceberg tiering support limits flexibility and
> ecosystem
> > > > integration
> > > > > for users who rely on Iceberg’s open table format.
> > > > >
> > > > > To address this gap, I’d like to propose FIP-3: Support
> Tiering
> > > Fluss
> > > > Data
> > > > > to Iceberg[1] which aims to integrate Iceberg into Fluss’s
> > > tiering
> > > > > capabilities.
> > > > >
> > > > > Welcome your feedback and suggestions on this proposal.
> Looking
> > > forward
> > > > to
> > > > > a productive discussion!
> > > > >
> > > > > [1]:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> > > >
> > > <
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg>
> > >;
> > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > >
> > > >
> >
>
Re: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by Mehul Batra <me...@gmail.com>.
Hi Yuxia,
Thanks for the clarification.
It's good to know that compaction/expiring-snapshots will be addressed in
the initial version, and I completely understand your point on the
complexity of orphan file cleanup. I agree it's better to avoid over-design
at this stage and evolve as needed based on future usage patterns and
feedback.
Also, thanks for confirming that snapshot expiration will be triggered
explicitly via the LakeCommitter. That clears things up for me.
Looking forward to working on this with you and the community, seeing how
this evolves!
Best regards,
Mehul
On Tue, Jul 8, 2025 at 7:16 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Hi, Mehul
>
> "Tableflow automates table maintenance by compacting and cleaning up small
> files generated by continuous streaming data in object storage."
> Seems table flow supports compacting, which is covered in this FIP.
> Haven't seen orphan file cleanup in table flow.
> Orphan file cleanup is not straightforward and a little of complex, which
> required to list all files, and compare with the files in iceberg manifest
> to find the orphan files.
> I still prefer not to introduce the complexity currently for the first
> version of iceberg support, which will cause overdesign. Let's just see in
> the future what'll happen.
>
> As for snapshot expiration. Yes, LakeCommitter should trigger the snapshot
> expiration action explicitly. It's a slight operation.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Mehul Batra" <me...@gmail.com>
> 收件人: "dev" <de...@fluss.apache.org>
> 发送时间: 星期二, 2025年 7 月 08日 上午 4:26:19
> 主题: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
>
> Hi Yuxia, Cheng,
>
> Thank you both for the insights.
>
> From a user’s perspective, I believe our goal should be to abstract away as
> much operational complexity as possible. For example, TableFlow handles
> both data writing and maintenance seamlessly for the user, which avoids the
> burden of running separate processes.
>
> https://docs.confluent.io/cloud/current/topics/tableflow/overview.html#table-maintenance-and-optimizations
>
> In Fluss Integration, if users are expected to run a separate maintenance
> job (e.g., for snapshot expiration or orphan file cleanup), there's a real
> risk of job overlap and failure, especially due to optimistic concurrency
> issues when both (tiering & maintenance) try to commit around the same
> time.
>
> Yuxia, you mentioned that the LakeCommitter will respect the
> history.expire.max-snapshot-age-ms property (similar to Paimon). I just
> wanted to clarify, while the property sets the retention policy, we still
> need to trigger the snapshot expiration action explicitly. Do we envision
> Fluss's tiering job playing that role?
>
> If so, that would be a great win, it could help automate snapshot
> expiration and indirectly clean up orphan files, making things much
> smoother for users.
>
> Please correct me if I’ve misunderstood anything.
>
> Best regards,
> Mehul
>
> On Mon, Jul 7, 2025 at 11:32 AM Wang Cheng <34...@qq.com.invalid>
> wrote:
>
> > Hi Mehul,
> >
> >
> > I agree with Yuxia's point. We should leave such table maintenance work
> > like expiring snapshots and deleting orphan files to Iceberg users
> > rather than relying on Fluss tiering job.
> >
> >
> >
> > Regards,
> > Cheng
> >
> >
> >
> >
> >
> >
> >
> >
> > ------------------ Original ------------------
> > From:
> > "dev"
> > <
> > mehulbatra1997@gmail.com>;
> > Date: Sat, Jul 5, 2025 11:38 PM
> > To: "dev"<dev@fluss.apache.org>;
> >
> > Subject: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> >
> >
> >
> > Hi Yuxia,
> > Great, that sounds good to me and will help the user to have a better
> read
> > latency.
> > How about the Snapshot expiration (to regulate metadata) and removing the
> > orphan files(which are no longer referenced or dangling files of failed
> > tasks)?
> > Are we planning to introduce them as part of automated maintenance
> provided
> > by the Fluss cluster?
> > Warm regards,
> > Mehul Batra
> >
> > On Fri, Jul 4, 2025 at 5:02 PM yuxia <luoyuxia@alumni.sjtu.edu.cn>
> > wrote:
> >
> > > Hi, Mehul.
> > > Thanks for your attention. I think we don't need to introduce an
> extra
> > > post-commit hook to manage small files. In the design, all files
> that
> > belong
> > > to same bucket(in iceberg, it'll be same partition) be distributed
> to
> > same
> > > task to write. So, the task can compact these small files then for
> the
> > > partition.
> > > As this FIP said, while creating IcebergLakeWriter in one round of
> > > tiering, the writer can scan manifest to know the files in this
> > bucket, if
> > > found compaction is available, it can
> > > compact these files while writing new files. We have a similar logic
> > for
> > > tiering to paimon.
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > ----- 原始邮件 -----
> > > 发件人: "Mehul Batra" <mehulbatra1997@gmail.com>
> > > 收件人: "dev" <dev@fluss.apache.org>
> > > 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> > > 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> > >
> > > +1 This will help us to address the missing table format and provide
> > better
> > > ecosystem interoperability. Iceberg's growing adoption in the data
> > > lakehouse space makes this a valuable addition to Fluss's tiering
> > > capabilities.
> > > Are there any plans to integrate the Maintenance services as part of
> > > tiering itself as a post-commit hook to manage small files?
> > > Warm regards,
> > > Mehul Batra
> > >
> > > On Thu, Jul 3, 2025 at 2:24 PM yuxia <luoyuxia@alumni.sjtu.edu.cn
> >
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Fluss currently supports tiering data to Apache Paimon,
> enabling
> > > > cost-effective storage management for warm/cold data. However,
> > the lack
> > > of
> > > > native Iceberg tiering support limits flexibility and ecosystem
> > > integration
> > > > for users who rely on Iceberg’s open table format.
> > > >
> > > > To address this gap, I’d like to propose FIP-3: Support Tiering
> > Fluss
> > > Data
> > > > to Iceberg[1] which aims to integrate Iceberg into Fluss’s
> > tiering
> > > > capabilities.
> > > >
> > > > Welcome your feedback and suggestions on this proposal. Looking
> > forward
> > > to
> > > > a productive discussion!
> > > >
> > > > [1]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> > >
> > <
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg>
> >;
> > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > >
> > >
>
Re: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, Mehul
"Tableflow automates table maintenance by compacting and cleaning up small files generated by continuous streaming data in object storage."
Seems table flow supports compacting, which is covered in this FIP. Haven't seen orphan file cleanup in table flow.
Orphan file cleanup is not straightforward and a little of complex, which required to list all files, and compare with the files in iceberg manifest to find the orphan files.
I still prefer not to introduce the complexity currently for the first version of iceberg support, which will cause overdesign. Let's just see in the future what'll happen.
As for snapshot expiration. Yes, LakeCommitter should trigger the snapshot expiration action explicitly. It's a slight operation.
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Mehul Batra" <me...@gmail.com>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期二, 2025年 7 月 08日 上午 4:26:19
主题: [SPAM]Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Hi Yuxia, Cheng,
Thank you both for the insights.
From a user’s perspective, I believe our goal should be to abstract away as
much operational complexity as possible. For example, TableFlow handles
both data writing and maintenance seamlessly for the user, which avoids the
burden of running separate processes.
https://docs.confluent.io/cloud/current/topics/tableflow/overview.html#table-maintenance-and-optimizations
In Fluss Integration, if users are expected to run a separate maintenance
job (e.g., for snapshot expiration or orphan file cleanup), there's a real
risk of job overlap and failure, especially due to optimistic concurrency
issues when both (tiering & maintenance) try to commit around the same time.
Yuxia, you mentioned that the LakeCommitter will respect the
history.expire.max-snapshot-age-ms property (similar to Paimon). I just
wanted to clarify, while the property sets the retention policy, we still
need to trigger the snapshot expiration action explicitly. Do we envision
Fluss's tiering job playing that role?
If so, that would be a great win, it could help automate snapshot
expiration and indirectly clean up orphan files, making things much
smoother for users.
Please correct me if I’ve misunderstood anything.
Best regards,
Mehul
On Mon, Jul 7, 2025 at 11:32 AM Wang Cheng <34...@qq.com.invalid> wrote:
> Hi Mehul,
>
>
> I agree with Yuxia's point. We should leave such table maintenance work
> like expiring snapshots and deleting orphan files to Iceberg users
> rather than relying on Fluss tiering job.
>
>
>
> Regards,
> Cheng
>
>
>
>
>
>
>
>
> ------------------ Original ------------------
> From:
> "dev"
> <
> mehulbatra1997@gmail.com>;
> Date: Sat, Jul 5, 2025 11:38 PM
> To: "dev"<dev@fluss.apache.org>;
>
> Subject: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
>
>
>
> Hi Yuxia,
> Great, that sounds good to me and will help the user to have a better read
> latency.
> How about the Snapshot expiration (to regulate metadata) and removing the
> orphan files(which are no longer referenced or dangling files of failed
> tasks)?
> Are we planning to introduce them as part of automated maintenance provided
> by the Fluss cluster?
> Warm regards,
> Mehul Batra
>
> On Fri, Jul 4, 2025 at 5:02 PM yuxia <luoyuxia@alumni.sjtu.edu.cn>
> wrote:
>
> > Hi, Mehul.
> > Thanks for your attention. I think we don't need to introduce an extra
> > post-commit hook to manage small files. In the design, all files that
> belong
> > to same bucket(in iceberg, it'll be same partition) be distributed to
> same
> > task to write. So, the task can compact these small files then for the
> > partition.
> > As this FIP said, while creating IcebergLakeWriter in one round of
> > tiering, the writer can scan manifest to know the files in this
> bucket, if
> > found compaction is available, it can
> > compact these files while writing new files. We have a similar logic
> for
> > tiering to paimon.
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Mehul Batra" <mehulbatra1997@gmail.com>
> > 收件人: "dev" <dev@fluss.apache.org>
> > 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> > 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> >
> > +1 This will help us to address the missing table format and provide
> better
> > ecosystem interoperability. Iceberg's growing adoption in the data
> > lakehouse space makes this a valuable addition to Fluss's tiering
> > capabilities.
> > Are there any plans to integrate the Maintenance services as part of
> > tiering itself as a post-commit hook to manage small files?
> > Warm regards,
> > Mehul Batra
> >
> > On Thu, Jul 3, 2025 at 2:24 PM yuxia <luoyuxia@alumni.sjtu.edu.cn>
> wrote:
> >
> > > Hi,
> > >
> > > Fluss currently supports tiering data to Apache Paimon, enabling
> > > cost-effective storage management for warm/cold data. However,
> the lack
> > of
> > > native Iceberg tiering support limits flexibility and ecosystem
> > integration
> > > for users who rely on Iceberg’s open table format.
> > >
> > > To address this gap, I’d like to propose FIP-3: Support Tiering
> Fluss
> > Data
> > > to Iceberg[1] which aims to integrate Iceberg into Fluss’s
> tiering
> > > capabilities.
> > >
> > > Welcome your feedback and suggestions on this proposal. Looking
> forward
> > to
> > > a productive discussion!
> > >
> > > [1]:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> >
> <https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg>>;
> >
> > > Best regards,
> > > Yuxia
> > >
> > >
> >
Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by Mehul Batra <me...@gmail.com>.
Hi Yuxia, Cheng,
Thank you both for the insights.
From a user’s perspective, I believe our goal should be to abstract away as
much operational complexity as possible. For example, TableFlow handles
both data writing and maintenance seamlessly for the user, which avoids the
burden of running separate processes.
https://docs.confluent.io/cloud/current/topics/tableflow/overview.html#table-maintenance-and-optimizations
In Fluss Integration, if users are expected to run a separate maintenance
job (e.g., for snapshot expiration or orphan file cleanup), there's a real
risk of job overlap and failure, especially due to optimistic concurrency
issues when both (tiering & maintenance) try to commit around the same time.
Yuxia, you mentioned that the LakeCommitter will respect the
history.expire.max-snapshot-age-ms property (similar to Paimon). I just
wanted to clarify, while the property sets the retention policy, we still
need to trigger the snapshot expiration action explicitly. Do we envision
Fluss's tiering job playing that role?
If so, that would be a great win, it could help automate snapshot
expiration and indirectly clean up orphan files, making things much
smoother for users.
Please correct me if I’ve misunderstood anything.
Best regards,
Mehul
On Mon, Jul 7, 2025 at 11:32 AM Wang Cheng <34...@qq.com.invalid> wrote:
> Hi Mehul,
>
>
> I agree with Yuxia's point. We should leave such table maintenance work
> like expiring snapshots and deleting orphan files to Iceberg users
> rather than relying on Fluss tiering job.
>
>
>
> Regards,
> Cheng
>
>
>
>
>
>
>
>
> ------------------ Original ------------------
> From:
> "dev"
> <
> mehulbatra1997@gmail.com>;
> Date: Sat, Jul 5, 2025 11:38 PM
> To: "dev"<dev@fluss.apache.org>;
>
> Subject: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
>
>
>
> Hi Yuxia,
> Great, that sounds good to me and will help the user to have a better read
> latency.
> How about the Snapshot expiration (to regulate metadata) and removing the
> orphan files(which are no longer referenced or dangling files of failed
> tasks)?
> Are we planning to introduce them as part of automated maintenance provided
> by the Fluss cluster?
> Warm regards,
> Mehul Batra
>
> On Fri, Jul 4, 2025 at 5:02 PM yuxia <luoyuxia@alumni.sjtu.edu.cn>
> wrote:
>
> > Hi, Mehul.
> > Thanks for your attention. I think we don't need to introduce an extra
> > post-commit hook to manage small files. In the design, all files that
> belong
> > to same bucket(in iceberg, it'll be same partition) be distributed to
> same
> > task to write. So, the task can compact these small files then for the
> > partition.
> > As this FIP said, while creating IcebergLakeWriter in one round of
> > tiering, the writer can scan manifest to know the files in this
> bucket, if
> > found compaction is available, it can
> > compact these files while writing new files. We have a similar logic
> for
> > tiering to paimon.
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Mehul Batra" <mehulbatra1997@gmail.com>
> > 收件人: "dev" <dev@fluss.apache.org>
> > 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> > 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
> >
> > +1 This will help us to address the missing table format and provide
> better
> > ecosystem interoperability. Iceberg's growing adoption in the data
> > lakehouse space makes this a valuable addition to Fluss's tiering
> > capabilities.
> > Are there any plans to integrate the Maintenance services as part of
> > tiering itself as a post-commit hook to manage small files?
> > Warm regards,
> > Mehul Batra
> >
> > On Thu, Jul 3, 2025 at 2:24 PM yuxia <luoyuxia@alumni.sjtu.edu.cn>
> wrote:
> >
> > > Hi,
> > >
> > > Fluss currently supports tiering data to Apache Paimon, enabling
> > > cost-effective storage management for warm/cold data. However,
> the lack
> > of
> > > native Iceberg tiering support limits flexibility and ecosystem
> > integration
> > > for users who rely on Iceberg’s open table format.
> > >
> > > To address this gap, I’d like to propose FIP-3: Support Tiering
> Fluss
> > Data
> > > to Iceberg[1] which aims to integrate Iceberg into Fluss’s
> tiering
> > > capabilities.
> > >
> > > Welcome your feedback and suggestions on this proposal. Looking
> forward
> > to
> > > a productive discussion!
> > >
> > > [1]:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> >
> <https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg>>;
> >
> > > Best regards,
> > > Yuxia
> > >
> > >
> >
Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by Wang Cheng <34...@qq.com.INVALID>.
Hi Mehul,
I agree with Yuxia's point. We should leave such table maintenance work like expiring snapshots and deleting orphan files to Iceberg users rather than relying on Fluss tiering job.
Regards,
Cheng
------------------ Original ------------------
From: "dev" <mehulbatra1997@gmail.com>;
Date: Sat, Jul 5, 2025 11:38 PM
To: "dev"<dev@fluss.apache.org>;
Subject: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Hi Yuxia,
Great, that sounds good to me and will help the user to have a better read
latency.
How about the Snapshot expiration (to regulate metadata) and removing the
orphan files(which are no longer referenced or dangling files of failed
tasks)?
Are we planning to introduce them as part of automated maintenance provided
by the Fluss cluster?
Warm regards,
Mehul Batra
On Fri, Jul 4, 2025 at 5:02 PM yuxia <luoyuxia@alumni.sjtu.edu.cn> wrote:
> Hi, Mehul.
> Thanks for your attention. I think we don't need to introduce an extra
> post-commit hook to manage small files. In the design, all files that belong
> to same bucket(in iceberg, it'll be same partition) be distributed to same
> task to write. So, the task can compact these small files then for the
> partition.
> As this FIP said, while creating IcebergLakeWriter in one round of
> tiering, the writer can scan manifest to know the files in this bucket, if
> found compaction is available, it can
> compact these files while writing new files. We have a similar logic for
> tiering to paimon.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Mehul Batra" <mehulbatra1997@gmail.com>
> 收件人: "dev" <dev@fluss.apache.org>
> 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
>
> +1 This will help us to address the missing table format and provide better
> ecosystem interoperability. Iceberg's growing adoption in the data
> lakehouse space makes this a valuable addition to Fluss's tiering
> capabilities.
> Are there any plans to integrate the Maintenance services as part of
> tiering itself as a post-commit hook to manage small files?
> Warm regards,
> Mehul Batra
>
> On Thu, Jul 3, 2025 at 2:24 PM yuxia <luoyuxia@alumni.sjtu.edu.cn> wrote:
>
> > Hi,
> >
> > Fluss currently supports tiering data to Apache Paimon, enabling
> > cost-effective storage management for warm/cold data. However, the lack
> of
> > native Iceberg tiering support limits flexibility and ecosystem
> integration
> > for users who rely on Iceberg’s open table format.
> >
> > To address this gap, I’d like to propose FIP-3: Support Tiering Fluss
> Data
> > to Iceberg[1] which aims to integrate Iceberg into Fluss’s tiering
> > capabilities.
> >
> > Welcome your feedback and suggestions on this proposal. Looking forward
> to
> > a productive discussion!
> >
> > [1]:
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> >
> > Best regards,
> > Yuxia
> >
> >
>
Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, Mehul.
For Snapshot expiration, as the FIP said, while committing by LakeCommitter, it should respect the iceberg table properties[1].
It will respect the property `history.expire.max-snapshot-age-ms` to expire snapshots while committing as we done for Paimon.
As for removing the orphan files, I feel like it'll be a little of complex for Fluss to do that. And also, I think it should be better triggered by user since it's hard for Fluss cluster
to know when to trigger removing the orphan files. So, let's leave it to user to see what happens.
[1]https://iceberg.apache.org/docs/1.9.1/configuration/#table-behavior-properties
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Mehul Batra" <me...@gmail.com>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期六, 2025年 7 月 05日 下午 11:38:20
主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Hi Yuxia,
Great, that sounds good to me and will help the user to have a better read
latency.
How about the Snapshot expiration (to regulate metadata) and removing the
orphan files(which are no longer referenced or dangling files of failed
tasks)?
Are we planning to introduce them as part of automated maintenance provided
by the Fluss cluster?
Warm regards,
Mehul Batra
On Fri, Jul 4, 2025 at 5:02 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Hi, Mehul.
> Thanks for your attention. I think we don't need to introduce an extra
> post-commit hook to manage small files. In the design, all files that belong
> to same bucket(in iceberg, it'll be same partition) be distributed to same
> task to write. So, the task can compact these small files then for the
> partition.
> As this FIP said, while creating IcebergLakeWriter in one round of
> tiering, the writer can scan manifest to know the files in this bucket, if
> found compaction is available, it can
> compact these files while writing new files. We have a similar logic for
> tiering to paimon.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Mehul Batra" <me...@gmail.com>
> 收件人: "dev" <de...@fluss.apache.org>
> 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
>
> +1 This will help us to address the missing table format and provide better
> ecosystem interoperability. Iceberg's growing adoption in the data
> lakehouse space makes this a valuable addition to Fluss's tiering
> capabilities.
> Are there any plans to integrate the Maintenance services as part of
> tiering itself as a post-commit hook to manage small files?
> Warm regards,
> Mehul Batra
>
> On Thu, Jul 3, 2025 at 2:24 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> > Hi,
> >
> > Fluss currently supports tiering data to Apache Paimon, enabling
> > cost-effective storage management for warm/cold data. However, the lack
> of
> > native Iceberg tiering support limits flexibility and ecosystem
> integration
> > for users who rely on Iceberg’s open table format.
> >
> > To address this gap, I’d like to propose FIP-3: Support Tiering Fluss
> Data
> > to Iceberg[1] which aims to integrate Iceberg into Fluss’s tiering
> > capabilities.
> >
> > Welcome your feedback and suggestions on this proposal. Looking forward
> to
> > a productive discussion!
> >
> > [1]:
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> >
> > Best regards,
> > Yuxia
> >
> >
>
Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by Mehul Batra <me...@gmail.com>.
Hi Yuxia,
Great, that sounds good to me and will help the user to have a better read
latency.
How about the Snapshot expiration (to regulate metadata) and removing the
orphan files(which are no longer referenced or dangling files of failed
tasks)?
Are we planning to introduce them as part of automated maintenance provided
by the Fluss cluster?
Warm regards,
Mehul Batra
On Fri, Jul 4, 2025 at 5:02 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Hi, Mehul.
> Thanks for your attention. I think we don't need to introduce an extra
> post-commit hook to manage small files. In the design, all files that belong
> to same bucket(in iceberg, it'll be same partition) be distributed to same
> task to write. So, the task can compact these small files then for the
> partition.
> As this FIP said, while creating IcebergLakeWriter in one round of
> tiering, the writer can scan manifest to know the files in this bucket, if
> found compaction is available, it can
> compact these files while writing new files. We have a similar logic for
> tiering to paimon.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Mehul Batra" <me...@gmail.com>
> 收件人: "dev" <de...@fluss.apache.org>
> 发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
> 主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
>
> +1 This will help us to address the missing table format and provide better
> ecosystem interoperability. Iceberg's growing adoption in the data
> lakehouse space makes this a valuable addition to Fluss's tiering
> capabilities.
> Are there any plans to integrate the Maintenance services as part of
> tiering itself as a post-commit hook to manage small files?
> Warm regards,
> Mehul Batra
>
> On Thu, Jul 3, 2025 at 2:24 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> > Hi,
> >
> > Fluss currently supports tiering data to Apache Paimon, enabling
> > cost-effective storage management for warm/cold data. However, the lack
> of
> > native Iceberg tiering support limits flexibility and ecosystem
> integration
> > for users who rely on Iceberg’s open table format.
> >
> > To address this gap, I’d like to propose FIP-3: Support Tiering Fluss
> Data
> > to Iceberg[1] which aims to integrate Iceberg into Fluss’s tiering
> > capabilities.
> >
> > Welcome your feedback and suggestions on this proposal. Looking forward
> to
> > a productive discussion!
> >
> > [1]:
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
> >
> > Best regards,
> > Yuxia
> >
> >
>
Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, Mehul.
Thanks for your attention. I think we don't need to introduce an extra post-commit hook to manage small files. In the design, all files that belong
to same bucket(in iceberg, it'll be same partition) be distributed to same task to write. So, the task can compact these small files then for the partition.
As this FIP said, while creating IcebergLakeWriter in one round of tiering, the writer can scan manifest to know the files in this bucket, if found compaction is available, it can
compact these files while writing new files. We have a similar logic for tiering to paimon.
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Mehul Batra" <me...@gmail.com>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期四, 2025年 7 月 03日 下午 5:04:18
主题: Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
+1 This will help us to address the missing table format and provide better
ecosystem interoperability. Iceberg's growing adoption in the data
lakehouse space makes this a valuable addition to Fluss's tiering
capabilities.
Are there any plans to integrate the Maintenance services as part of
tiering itself as a post-commit hook to manage small files?
Warm regards,
Mehul Batra
On Thu, Jul 3, 2025 at 2:24 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Hi,
>
> Fluss currently supports tiering data to Apache Paimon, enabling
> cost-effective storage management for warm/cold data. However, the lack of
> native Iceberg tiering support limits flexibility and ecosystem integration
> for users who rely on Iceberg’s open table format.
>
> To address this gap, I’d like to propose FIP-3: Support Tiering Fluss Data
> to Iceberg[1] which aims to integrate Iceberg into Fluss’s tiering
> capabilities.
>
> Welcome your feedback and suggestions on this proposal. Looking forward to
> a productive discussion!
>
> [1]:
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
>
> Best regards,
> Yuxia
>
>
Re: [DISCUSS] FIP-3: Support tiering Fluss data to Iceberg
Posted by Mehul Batra <me...@gmail.com>.
+1 This will help us to address the missing table format and provide better
ecosystem interoperability. Iceberg's growing adoption in the data
lakehouse space makes this a valuable addition to Fluss's tiering
capabilities.
Are there any plans to integrate the Maintenance services as part of
tiering itself as a post-commit hook to manage small files?
Warm regards,
Mehul Batra
On Thu, Jul 3, 2025 at 2:24 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> Hi,
>
> Fluss currently supports tiering data to Apache Paimon, enabling
> cost-effective storage management for warm/cold data. However, the lack of
> native Iceberg tiering support limits flexibility and ecosystem integration
> for users who rely on Iceberg’s open table format.
>
> To address this gap, I’d like to propose FIP-3: Support Tiering Fluss Data
> to Iceberg[1] which aims to integrate Iceberg into Fluss’s tiering
> capabilities.
>
> Welcome your feedback and suggestions on this proposal. Looking forward to
> a productive discussion!
>
> [1]:
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-3%3A+Support+tiering+Fluss+data+to+Iceberg
>
> Best regards,
> Yuxia
>
>