You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Andy Grove <an...@gmail.com> on 2026/03/18 12:23:37 UTC

Re: [DISCUSS] Future of Iceberg support in Apache DataFusion's Comet Spark Accelerator

Hi,

The Comet community has made good progress with the iceberg-rust
integration and we now believe that this is the best path forward for
Comet, so are no longer going to be working on the integration with
Iceberg-Java. I posted a comment on the discussion issue as well [1].

Thanks for the community support with all of this. We will follow up with a
PR soon to remove the current (broken) Comet integration from Iceberg-Java.

Thanks,

Andy.

[1]
https://github.com/apache/datafusion-comet/issues/2921#issuecomment-4048408537

On Fri, Feb 20, 2026 at 7:03 AM Péter Váry <pe...@gmail.com>
wrote:

> Hi Team,
> The File Format API is merged, so that should not block the progress on
> the Comet integration progress anymore.
> If you need help, let me know.
> Thanks,
> Peter
>
> Andy Grove <an...@gmail.com> ezt írta (időpont: 2026. jan. 22., Cs,
> 21:51):
>
>> I added another comment [1] on the issue [2], but will share here as well
>> for maximum visibility.
>>
>> I have two PRs in Comet that I am looking for reviews on.
>>
>> The first PR adds an @IcebergApi annotation to all Java
>> classes/methods/fields that are currently referenced from the Iceberg main
>> branch, and also adds documentation to the contributors guide.
>> #3237 <https://github.com/apache/datafusion-comet/pull/3237>
>>
>> The second PR builds on this and adds new dedicated unit tests for API
>> stability. This PR is more consequential because it discusses
>> deprecating/removing Comet's native_comet scan and how that relates to
>> the Iceberg API.
>> #3240 <https://github.com/apache/datafusion-comet/pull/3240>
>>
>> I am not an expert on the current integration, so it is possible that I
>> may have misunderstood things. I would appreciate reviews from both the
>> Iceberg and Comet communities!
>> Thanks,
>>
>> Andy.
>>
>> [1]
>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915
>> [2] https://github.com/apache/datafusion-comet/issues/2921
>>
>> On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <ke...@apache.org> wrote:
>>
>>> Thanks for starting the thread! I've added a comment on the Github
>>> issue. Super exciting to see the iceberg-rust integration with Comet.
>>> As mentioned, I'll help take a look at
>>> https://github.com/apache/iceberg/pull/13786 and see if we can bring it
>>> to the finish line.
>>>
>>> Best,
>>> Kevin Liu
>>>
>>> On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <li...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Andy for raising this.
>>>>
>>>> +1 for the iceberg-rust solution. This could benefit the whole oss
>>>> community in the long term and avoid duplicated efforts. As with the
>>>> technique challenges you mentioned, this seems more like a missing feature
>>>> rather than limitation.
>>>>
>>>> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <hu...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Andy for starting this discussion!
>>>>>
>>>>> I agree we should prioritize the iceberg-rust path as the long term
>>>>> default for Comet/Iceberg scans, while it's still valuable to keep the
>>>>> Iceberg Java integration as an option, especially for users who need
>>>>> JVM-specific features. I'm hoping we can get consensus on a path to resolve
>>>>> the shading issues (e.g. the approach proposed in
>>>>> github.com/apache/iceberg/pull/13786) so the java option can move
>>>>> forward.
>>>>>
>>>>> Thanks,
>>>>> Huaxin
>>>>>
>>>>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <ye...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Andy for starting this thread!
>>>>>>
>>>>>> I've replied to the issue above but am posting here as well for
>>>>>> better visibility:
>>>>>>
>>>>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the
>>>>>> shading and circular dependency headaches of the Java approach. Easier
>>>>>> maintenance typically lowers the entry barrier and encourages more
>>>>>> community involvement, which is important for the project’s long-term
>>>>>> health.
>>>>>>
>>>>>> It also makes sense to keep the Iceberg Java integration available as
>>>>>> an experimental option for users who need its richer JVM-specific features
>>>>>> while Rust becomes the long-term default. We don’t want to block
>>>>>> contributions from anyone who truly needs the Java path.
>>>>>>
>>>>>> Best,
>>>>>> Shawn
>>>>>>
>>>>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> There is a discussion in the DataFusion Comet community about the
>>>>>>> future direction of our Iceberg support [1]. For example, should we
>>>>>>> continue the efforts to integrate with the Iceberg Java project or focus
>>>>>>> more on the iceberg-rust project.
>>>>>>>
>>>>>>> It would be great to also get input from the Iceberg community.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Andy.
>>>>>>>
>>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921
>>>>>>>
>>>>>>

Re: [DISCUSS] Future of Iceberg support in Apache DataFusion's Comet Spark Accelerator

Posted by Anurag Mantripragada <an...@gmail.com>.
Hi Parth and Andy,

Thank you both for the context. I will add this topic to the agenda for the
next Iceberg community sync to discuss the pending work and determine the
best way forward.

~ Anurag

On Fri, Mar 20, 2026 at 3:18 PM Parth Chandra <pa...@apache.org> wrote:

> The PR was closed because it required the Comet code that has now been
> removed. Even with that code reinstated, the PR is out of date and will
> likely need to be updated to make it work with the latest code base. There
> is also a PR for deleted rows support [1] which will also need to be
> updated.
>
> [1] https://github.com/apache/iceberg/pull/14062
>
> On Fri, Mar 20, 2026 at 10:50 AM Andy Grove <an...@gmail.com> wrote:
>
>> Hi Anurag,
>>
>> I have no objection to my PR being reverted to allow for more
>> discussion. I think that would make sense.
>>
>> My only concern is that I am not sure how we would "fix the broken
>> code". My understanding is that there was a PR that aimed to do that
>> [1] but it was just closed after 7 months and 98 comments. Parth can
>> likely add more context here though.
>>
>> I'm open to suggestions on the best way to handle this.
>>
>> Thanks,
>>
>> Andy.
>>
>>
>> [1] https://github.com/apache/iceberg/pull/13786
>>
>>
>> On Fri, Mar 20, 2026 at 11:28 AM Anurag Mantripragada
>> <an...@apache.org> wrote:
>> >
>> > Thanks for the PR, Andy.
>> >
>> > I see that it has been merged, but I feel we haven't yet had the
>> opportunity to discuss this in detail within the Iceberg community. Kevin
>> and I were discussing this offline and feel that removing the broken code
>> in its current state makes it difficult to restore in the future.
>> >
>> > I propose the following:
>> > - Revert the removal PR.
>> > - Fix the broken code and include it in a patch release so we freeze a
>> working version.
>> > - Discuss the removal during the next Iceberg community sync on April 1.
>> > - If we reach a consensus, re-apply the removal PR.
>> >
>> > What do you think?
>> >
>> > ~ Anurag
>> >
>> > On Wed, Mar 18, 2026 at 9:54 AM Andy Grove <an...@gmail.com>
>> wrote:
>> >>
>> >> I have a draft PR up to remove the current Comet integration. I am
>> waiting for CI to finish before marking it as ready for review.
>> >>
>> >> https://github.com/apache/iceberg/pull/15674
>> >>
>> >> On Wed, Mar 18, 2026 at 10:43 AM Kevin Liu <ke...@apache.org>
>> wrote:
>> >>>
>> >>> Thanks for the update, Andy!
>> >>> It looks like most of the Comet code lives in the Spark integration
>> [1]. We're currently working on the next 1.11 release. I know the removal
>> isn't time-sensitive, but just wanted to mention it.
>> >>>
>> >>> Best,
>> >>> Kevin Liu
>> >>>
>> >>> [1]
>> https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet
>> >>>
>> >>> On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <an...@gmail.com>
>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> The Comet community has made good progress with the iceberg-rust
>> integration and we now believe that this is the best path forward for
>> Comet, so are no longer going to be working on the integration with
>> Iceberg-Java. I posted a comment on the discussion issue as well [1].
>> >>>>
>> >>>> Thanks for the community support with all of this. We will follow up
>> with a PR soon to remove the current (broken) Comet integration from
>> Iceberg-Java.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Andy.
>>
>

Re: [DISCUSS] Future of Iceberg support in Apache DataFusion's Comet Spark Accelerator

Posted by Parth Chandra <pa...@apache.org>.
The PR was closed because it required the Comet code that has now been
removed. Even with that code reinstated, the PR is out of date and will
likely need to be updated to make it work with the latest code base. There
is also a PR for deleted rows support [1] which will also need to be
updated.

[1] https://github.com/apache/iceberg/pull/14062

On Fri, Mar 20, 2026 at 10:50 AM Andy Grove <an...@gmail.com> wrote:

> Hi Anurag,
>
> I have no objection to my PR being reverted to allow for more
> discussion. I think that would make sense.
>
> My only concern is that I am not sure how we would "fix the broken
> code". My understanding is that there was a PR that aimed to do that
> [1] but it was just closed after 7 months and 98 comments. Parth can
> likely add more context here though.
>
> I'm open to suggestions on the best way to handle this.
>
> Thanks,
>
> Andy.
>
>
> [1] https://github.com/apache/iceberg/pull/13786
>
>
> On Fri, Mar 20, 2026 at 11:28 AM Anurag Mantripragada
> <an...@apache.org> wrote:
> >
> > Thanks for the PR, Andy.
> >
> > I see that it has been merged, but I feel we haven't yet had the
> opportunity to discuss this in detail within the Iceberg community. Kevin
> and I were discussing this offline and feel that removing the broken code
> in its current state makes it difficult to restore in the future.
> >
> > I propose the following:
> > - Revert the removal PR.
> > - Fix the broken code and include it in a patch release so we freeze a
> working version.
> > - Discuss the removal during the next Iceberg community sync on April 1.
> > - If we reach a consensus, re-apply the removal PR.
> >
> > What do you think?
> >
> > ~ Anurag
> >
> > On Wed, Mar 18, 2026 at 9:54 AM Andy Grove <an...@gmail.com>
> wrote:
> >>
> >> I have a draft PR up to remove the current Comet integration. I am
> waiting for CI to finish before marking it as ready for review.
> >>
> >> https://github.com/apache/iceberg/pull/15674
> >>
> >> On Wed, Mar 18, 2026 at 10:43 AM Kevin Liu <ke...@apache.org>
> wrote:
> >>>
> >>> Thanks for the update, Andy!
> >>> It looks like most of the Comet code lives in the Spark integration
> [1]. We're currently working on the next 1.11 release. I know the removal
> isn't time-sensitive, but just wanted to mention it.
> >>>
> >>> Best,
> >>> Kevin Liu
> >>>
> >>> [1]
> https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet
> >>>
> >>> On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <an...@gmail.com>
> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> The Comet community has made good progress with the iceberg-rust
> integration and we now believe that this is the best path forward for
> Comet, so are no longer going to be working on the integration with
> Iceberg-Java. I posted a comment on the discussion issue as well [1].
> >>>>
> >>>> Thanks for the community support with all of this. We will follow up
> with a PR soon to remove the current (broken) Comet integration from
> Iceberg-Java.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Andy.
>

Re: [DISCUSS] Future of Iceberg support in Apache DataFusion's Comet Spark Accelerator

Posted by Andy Grove <an...@gmail.com>.
Hi Anurag,

I have no objection to my PR being reverted to allow for more
discussion. I think that would make sense.

My only concern is that I am not sure how we would "fix the broken
code". My understanding is that there was a PR that aimed to do that
[1] but it was just closed after 7 months and 98 comments. Parth can
likely add more context here though.

I'm open to suggestions on the best way to handle this.

Thanks,

Andy.


[1] https://github.com/apache/iceberg/pull/13786





On Fri, Mar 20, 2026 at 11:28 AM Anurag Mantripragada
<an...@apache.org> wrote:
>
> Thanks for the PR, Andy.
>
> I see that it has been merged, but I feel we haven't yet had the opportunity to discuss this in detail within the Iceberg community. Kevin and I were discussing this offline and feel that removing the broken code in its current state makes it difficult to restore in the future.
>
> I propose the following:
> - Revert the removal PR.
> - Fix the broken code and include it in a patch release so we freeze a working version.
> - Discuss the removal during the next Iceberg community sync on April 1.
> - If we reach a consensus, re-apply the removal PR.
>
> What do you think?
>
> ~ Anurag
>
> On Wed, Mar 18, 2026 at 9:54 AM Andy Grove <an...@gmail.com> wrote:
>>
>> I have a draft PR up to remove the current Comet integration. I am waiting for CI to finish before marking it as ready for review.
>>
>> https://github.com/apache/iceberg/pull/15674
>>
>> On Wed, Mar 18, 2026 at 10:43 AM Kevin Liu <ke...@apache.org> wrote:
>>>
>>> Thanks for the update, Andy!
>>> It looks like most of the Comet code lives in the Spark integration [1]. We're currently working on the next 1.11 release. I know the removal isn't time-sensitive, but just wanted to mention it.
>>>
>>> Best,
>>> Kevin Liu
>>>
>>> [1] https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet
>>>
>>> On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <an...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> The Comet community has made good progress with the iceberg-rust integration and we now believe that this is the best path forward for Comet, so are no longer going to be working on the integration with Iceberg-Java. I posted a comment on the discussion issue as well [1].
>>>>
>>>> Thanks for the community support with all of this. We will follow up with a PR soon to remove the current (broken) Comet integration from Iceberg-Java.
>>>>
>>>> Thanks,
>>>>
>>>> Andy.
>>>>
>>>> [1] https://github.com/apache/datafusion-comet/issues/2921#issuecomment-4048408537
>>>>
>>>> On Fri, Feb 20, 2026 at 7:03 AM Péter Váry <pe...@gmail.com> wrote:
>>>>>
>>>>> Hi Team,
>>>>> The File Format API is merged, so that should not block the progress on the Comet integration progress anymore.
>>>>> If you need help, let me know.
>>>>> Thanks,
>>>>> Peter
>>>>>
>>>>> Andy Grove <an...@gmail.com> ezt írta (időpont: 2026. jan. 22., Cs, 21:51):
>>>>>>
>>>>>> I added another comment [1] on the issue [2], but will share here as well for maximum visibility.
>>>>>>
>>>>>> I have two PRs in Comet that I am looking for reviews on.
>>>>>>
>>>>>> The first PR adds an @IcebergApi annotation to all Java classes/methods/fields that are currently referenced from the Iceberg main branch, and also adds documentation to the contributors guide.
>>>>>> #3237
>>>>>>
>>>>>> The second PR builds on this and adds new dedicated unit tests for API stability. This PR is more consequential because it discusses deprecating/removing Comet's native_comet scan and how that relates to the Iceberg API.
>>>>>> #3240
>>>>>>
>>>>>> I am not an expert on the current integration, so it is possible that I may have misunderstood things. I would appreciate reviews from both the Iceberg and Comet communities!
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Andy.
>>>>>>
>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915
>>>>>> [2] https://github.com/apache/datafusion-comet/issues/2921
>>>>>>
>>>>>> On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <ke...@apache.org> wrote:
>>>>>>>
>>>>>>> Thanks for starting the thread! I've added a comment on the Github issue. Super exciting to see the iceberg-rust integration with Comet.
>>>>>>> As mentioned, I'll help take a look at https://github.com/apache/iceberg/pull/13786 and see if we can bring it to the finish line.
>>>>>>>
>>>>>>> Best,
>>>>>>> Kevin Liu
>>>>>>>
>>>>>>> On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <li...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Thanks Andy for raising this.
>>>>>>>>
>>>>>>>> +1 for the iceberg-rust solution. This could benefit the whole oss community in the long term and avoid duplicated efforts. As with the technique challenges you mentioned, this seems more like a missing feature rather than limitation.
>>>>>>>>
>>>>>>>> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <hu...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Thanks Andy for starting this discussion!
>>>>>>>>>
>>>>>>>>> I agree we should prioritize the iceberg-rust path as the long term default for Comet/Iceberg scans, while it's still valuable to keep the Iceberg Java integration as an option, especially for users who need JVM-specific features. I'm hoping we can get consensus on a path to resolve the shading issues (e.g. the approach proposed in github.com/apache/iceberg/pull/13786) so the java option can move forward.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Huaxin
>>>>>>>>>
>>>>>>>>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <ye...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks Andy for starting this thread!
>>>>>>>>>>
>>>>>>>>>> I've replied to the issue above but am posting here as well for better visibility:
>>>>>>>>>>
>>>>>>>>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the shading and circular dependency headaches of the Java approach. Easier maintenance typically lowers the entry barrier and encourages more community involvement, which is important for the project’s long-term health.
>>>>>>>>>>
>>>>>>>>>> It also makes sense to keep the Iceberg Java integration available as an experimental option for users who need its richer JVM-specific features while Rust becomes the long-term default. We don’t want to block contributions from anyone who truly needs the Java path.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Shawn
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <an...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> There is a discussion in the DataFusion Comet community about the future direction of our Iceberg support [1]. For example, should we continue the efforts to integrate with the Iceberg Java project or focus more on the iceberg-rust project.
>>>>>>>>>>>
>>>>>>>>>>> It would be great to also get input from the Iceberg community.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Andy.
>>>>>>>>>>>
>>>>>>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921

Re: [DISCUSS] Future of Iceberg support in Apache DataFusion's Comet Spark Accelerator

Posted by Anurag Mantripragada <an...@apache.org>.
Thanks for the PR, Andy.

I see that it has been merged, but I feel we haven't yet had the
opportunity to discuss this in detail within the Iceberg community. Kevin
and I were discussing this offline and feel that removing the broken code
in its current state makes it difficult to restore in the future.

I propose the following:
- Revert the removal PR.
- Fix the broken code and include it in a patch release so we freeze a
working version.
- Discuss the removal during the next Iceberg community sync on April 1.
- If we reach a consensus, re-apply the removal PR.

What do you think?

~ Anurag

On Wed, Mar 18, 2026 at 9:54 AM Andy Grove <an...@gmail.com> wrote:

> I have a draft PR up to remove the current Comet integration. I am
> waiting for CI to finish before marking it as ready for review.
>
> https://github.com/apache/iceberg/pull/15674
>
> On Wed, Mar 18, 2026 at 10:43 AM Kevin Liu <ke...@apache.org> wrote:
>
>> Thanks for the update, Andy!
>> It looks like most of the Comet code lives in the Spark integration [1].
>> We're currently working on the next 1.11 release. I know the removal isn't
>> time-sensitive, but just wanted to mention it.
>>
>> Best,
>> Kevin Liu
>>
>> [1]
>> https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet
>>
>> On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <an...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> The Comet community has made good progress with the iceberg-rust
>>> integration and we now believe that this is the best path forward for
>>> Comet, so are no longer going to be working on the integration with
>>> Iceberg-Java. I posted a comment on the discussion issue as well [1].
>>>
>>> Thanks for the community support with all of this. We will follow up
>>> with a PR soon to remove the current (broken) Comet integration from
>>> Iceberg-Java.
>>>
>>> Thanks,
>>>
>>> Andy.
>>>
>>> [1]
>>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-4048408537
>>>
>>> On Fri, Feb 20, 2026 at 7:03 AM Péter Váry <pe...@gmail.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>> The File Format API is merged, so that should not block the progress on
>>>> the Comet integration progress anymore.
>>>> If you need help, let me know.
>>>> Thanks,
>>>> Peter
>>>>
>>>> Andy Grove <an...@gmail.com> ezt írta (időpont: 2026. jan. 22.,
>>>> Cs, 21:51):
>>>>
>>>>> I added another comment [1] on the issue [2], but will share here as
>>>>> well for maximum visibility.
>>>>>
>>>>> I have two PRs in Comet that I am looking for reviews on.
>>>>>
>>>>> The first PR adds an @IcebergApi annotation to all Java
>>>>> classes/methods/fields that are currently referenced from the Iceberg main
>>>>> branch, and also adds documentation to the contributors guide.
>>>>> #3237 <https://github.com/apache/datafusion-comet/pull/3237>
>>>>>
>>>>> The second PR builds on this and adds new dedicated unit tests for API
>>>>> stability. This PR is more consequential because it discusses
>>>>> deprecating/removing Comet's native_comet scan and how that relates
>>>>> to the Iceberg API.
>>>>> #3240 <https://github.com/apache/datafusion-comet/pull/3240>
>>>>>
>>>>> I am not an expert on the current integration, so it is possible that
>>>>> I may have misunderstood things. I would appreciate reviews from both the
>>>>> Iceberg and Comet communities!
>>>>> Thanks,
>>>>>
>>>>> Andy.
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915
>>>>> [2] https://github.com/apache/datafusion-comet/issues/2921
>>>>>
>>>>> On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <ke...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thanks for starting the thread! I've added a comment on the Github
>>>>>> issue. Super exciting to see the iceberg-rust integration with Comet.
>>>>>> As mentioned, I'll help take a look at
>>>>>> https://github.com/apache/iceberg/pull/13786 and see if we can bring
>>>>>> it to the finish line.
>>>>>>
>>>>>> Best,
>>>>>> Kevin Liu
>>>>>>
>>>>>> On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <li...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Andy for raising this.
>>>>>>>
>>>>>>> +1 for the iceberg-rust solution. This could benefit the whole oss
>>>>>>> community in the long term and avoid duplicated efforts. As with the
>>>>>>> technique challenges you mentioned, this seems more like a missing feature
>>>>>>> rather than limitation.
>>>>>>>
>>>>>>> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <hu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Andy for starting this discussion!
>>>>>>>>
>>>>>>>> I agree we should prioritize the iceberg-rust path as the long term
>>>>>>>> default for Comet/Iceberg scans, while it's still valuable to keep the
>>>>>>>> Iceberg Java integration as an option, especially for users who need
>>>>>>>> JVM-specific features. I'm hoping we can get consensus on a path to resolve
>>>>>>>> the shading issues (e.g. the approach proposed in
>>>>>>>> github.com/apache/iceberg/pull/13786) so the java option can move
>>>>>>>> forward.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Huaxin
>>>>>>>>
>>>>>>>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <
>>>>>>>> yexiangchang@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Andy for starting this thread!
>>>>>>>>>
>>>>>>>>> I've replied to the issue above but am posting here as well for
>>>>>>>>> better visibility:
>>>>>>>>>
>>>>>>>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the
>>>>>>>>> shading and circular dependency headaches of the Java approach. Easier
>>>>>>>>> maintenance typically lowers the entry barrier and encourages more
>>>>>>>>> community involvement, which is important for the project’s long-term
>>>>>>>>> health.
>>>>>>>>>
>>>>>>>>> It also makes sense to keep the Iceberg Java integration available
>>>>>>>>> as an experimental option for users who need its richer JVM-specific
>>>>>>>>> features while Rust becomes the long-term default. We don’t want to block
>>>>>>>>> contributions from anyone who truly needs the Java path.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Shawn
>>>>>>>>>
>>>>>>>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <an...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> There is a discussion in the DataFusion Comet community about the
>>>>>>>>>> future direction of our Iceberg support [1]. For example, should we
>>>>>>>>>> continue the efforts to integrate with the Iceberg Java project or focus
>>>>>>>>>> more on the iceberg-rust project.
>>>>>>>>>>
>>>>>>>>>> It would be great to also get input from the Iceberg community.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Andy.
>>>>>>>>>>
>>>>>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921
>>>>>>>>>>
>>>>>>>>>

Re: [DISCUSS] Future of Iceberg support in Apache DataFusion's Comet Spark Accelerator

Posted by Andy Grove <an...@gmail.com>.
I have a draft PR up to remove the current Comet integration. I am
waiting for CI to finish before marking it as ready for review.

https://github.com/apache/iceberg/pull/15674

On Wed, Mar 18, 2026 at 10:43 AM Kevin Liu <ke...@apache.org> wrote:

> Thanks for the update, Andy!
> It looks like most of the Comet code lives in the Spark integration [1].
> We're currently working on the next 1.11 release. I know the removal isn't
> time-sensitive, but just wanted to mention it.
>
> Best,
> Kevin Liu
>
> [1]
> https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet
>
> On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <an...@gmail.com> wrote:
>
>> Hi,
>>
>> The Comet community has made good progress with the iceberg-rust
>> integration and we now believe that this is the best path forward for
>> Comet, so are no longer going to be working on the integration with
>> Iceberg-Java. I posted a comment on the discussion issue as well [1].
>>
>> Thanks for the community support with all of this. We will follow up with
>> a PR soon to remove the current (broken) Comet integration from
>> Iceberg-Java.
>>
>> Thanks,
>>
>> Andy.
>>
>> [1]
>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-4048408537
>>
>> On Fri, Feb 20, 2026 at 7:03 AM Péter Váry <pe...@gmail.com>
>> wrote:
>>
>>> Hi Team,
>>> The File Format API is merged, so that should not block the progress on
>>> the Comet integration progress anymore.
>>> If you need help, let me know.
>>> Thanks,
>>> Peter
>>>
>>> Andy Grove <an...@gmail.com> ezt írta (időpont: 2026. jan. 22.,
>>> Cs, 21:51):
>>>
>>>> I added another comment [1] on the issue [2], but will share here as
>>>> well for maximum visibility.
>>>>
>>>> I have two PRs in Comet that I am looking for reviews on.
>>>>
>>>> The first PR adds an @IcebergApi annotation to all Java
>>>> classes/methods/fields that are currently referenced from the Iceberg main
>>>> branch, and also adds documentation to the contributors guide.
>>>> #3237 <https://github.com/apache/datafusion-comet/pull/3237>
>>>>
>>>> The second PR builds on this and adds new dedicated unit tests for API
>>>> stability. This PR is more consequential because it discusses
>>>> deprecating/removing Comet's native_comet scan and how that relates to
>>>> the Iceberg API.
>>>> #3240 <https://github.com/apache/datafusion-comet/pull/3240>
>>>>
>>>> I am not an expert on the current integration, so it is possible that I
>>>> may have misunderstood things. I would appreciate reviews from both the
>>>> Iceberg and Comet communities!
>>>> Thanks,
>>>>
>>>> Andy.
>>>>
>>>> [1]
>>>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915
>>>> [2] https://github.com/apache/datafusion-comet/issues/2921
>>>>
>>>> On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <ke...@apache.org>
>>>> wrote:
>>>>
>>>>> Thanks for starting the thread! I've added a comment on the Github
>>>>> issue. Super exciting to see the iceberg-rust integration with Comet.
>>>>> As mentioned, I'll help take a look at
>>>>> https://github.com/apache/iceberg/pull/13786 and see if we can bring
>>>>> it to the finish line.
>>>>>
>>>>> Best,
>>>>> Kevin Liu
>>>>>
>>>>> On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <li...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Andy for raising this.
>>>>>>
>>>>>> +1 for the iceberg-rust solution. This could benefit the whole oss
>>>>>> community in the long term and avoid duplicated efforts. As with the
>>>>>> technique challenges you mentioned, this seems more like a missing feature
>>>>>> rather than limitation.
>>>>>>
>>>>>> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <hu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Andy for starting this discussion!
>>>>>>>
>>>>>>> I agree we should prioritize the iceberg-rust path as the long term
>>>>>>> default for Comet/Iceberg scans, while it's still valuable to keep the
>>>>>>> Iceberg Java integration as an option, especially for users who need
>>>>>>> JVM-specific features. I'm hoping we can get consensus on a path to resolve
>>>>>>> the shading issues (e.g. the approach proposed in
>>>>>>> github.com/apache/iceberg/pull/13786) so the java option can move
>>>>>>> forward.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Huaxin
>>>>>>>
>>>>>>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <ye...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Andy for starting this thread!
>>>>>>>>
>>>>>>>> I've replied to the issue above but am posting here as well for
>>>>>>>> better visibility:
>>>>>>>>
>>>>>>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the
>>>>>>>> shading and circular dependency headaches of the Java approach. Easier
>>>>>>>> maintenance typically lowers the entry barrier and encourages more
>>>>>>>> community involvement, which is important for the project’s long-term
>>>>>>>> health.
>>>>>>>>
>>>>>>>> It also makes sense to keep the Iceberg Java integration available
>>>>>>>> as an experimental option for users who need its richer JVM-specific
>>>>>>>> features while Rust becomes the long-term default. We don’t want to block
>>>>>>>> contributions from anyone who truly needs the Java path.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Shawn
>>>>>>>>
>>>>>>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <an...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> There is a discussion in the DataFusion Comet community about the
>>>>>>>>> future direction of our Iceberg support [1]. For example, should we
>>>>>>>>> continue the efforts to integrate with the Iceberg Java project or focus
>>>>>>>>> more on the iceberg-rust project.
>>>>>>>>>
>>>>>>>>> It would be great to also get input from the Iceberg community.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Andy.
>>>>>>>>>
>>>>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921
>>>>>>>>>
>>>>>>>>

Re: [DISCUSS] Future of Iceberg support in Apache DataFusion's Comet Spark Accelerator

Posted by Kevin Liu <ke...@apache.org>.
Thanks for the update, Andy!
It looks like most of the Comet code lives in the Spark integration [1].
We're currently working on the next 1.11 release. I know the removal isn't
time-sensitive, but just wanted to mention it.

Best,
Kevin Liu

[1]
https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet

On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <an...@gmail.com> wrote:

> Hi,
>
> The Comet community has made good progress with the iceberg-rust
> integration and we now believe that this is the best path forward for
> Comet, so are no longer going to be working on the integration with
> Iceberg-Java. I posted a comment on the discussion issue as well [1].
>
> Thanks for the community support with all of this. We will follow up with
> a PR soon to remove the current (broken) Comet integration from
> Iceberg-Java.
>
> Thanks,
>
> Andy.
>
> [1]
> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-4048408537
>
> On Fri, Feb 20, 2026 at 7:03 AM Péter Váry <pe...@gmail.com>
> wrote:
>
>> Hi Team,
>> The File Format API is merged, so that should not block the progress on
>> the Comet integration progress anymore.
>> If you need help, let me know.
>> Thanks,
>> Peter
>>
>> Andy Grove <an...@gmail.com> ezt írta (időpont: 2026. jan. 22.,
>> Cs, 21:51):
>>
>>> I added another comment [1] on the issue [2], but will share here as
>>> well for maximum visibility.
>>>
>>> I have two PRs in Comet that I am looking for reviews on.
>>>
>>> The first PR adds an @IcebergApi annotation to all Java
>>> classes/methods/fields that are currently referenced from the Iceberg main
>>> branch, and also adds documentation to the contributors guide.
>>> #3237 <https://github.com/apache/datafusion-comet/pull/3237>
>>>
>>> The second PR builds on this and adds new dedicated unit tests for API
>>> stability. This PR is more consequential because it discusses
>>> deprecating/removing Comet's native_comet scan and how that relates to
>>> the Iceberg API.
>>> #3240 <https://github.com/apache/datafusion-comet/pull/3240>
>>>
>>> I am not an expert on the current integration, so it is possible that I
>>> may have misunderstood things. I would appreciate reviews from both the
>>> Iceberg and Comet communities!
>>> Thanks,
>>>
>>> Andy.
>>>
>>> [1]
>>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915
>>> [2] https://github.com/apache/datafusion-comet/issues/2921
>>>
>>> On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <ke...@apache.org> wrote:
>>>
>>>> Thanks for starting the thread! I've added a comment on the Github
>>>> issue. Super exciting to see the iceberg-rust integration with Comet.
>>>> As mentioned, I'll help take a look at
>>>> https://github.com/apache/iceberg/pull/13786 and see if we can bring
>>>> it to the finish line.
>>>>
>>>> Best,
>>>> Kevin Liu
>>>>
>>>> On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <li...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Andy for raising this.
>>>>>
>>>>> +1 for the iceberg-rust solution. This could benefit the whole oss
>>>>> community in the long term and avoid duplicated efforts. As with the
>>>>> technique challenges you mentioned, this seems more like a missing feature
>>>>> rather than limitation.
>>>>>
>>>>> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <hu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Andy for starting this discussion!
>>>>>>
>>>>>> I agree we should prioritize the iceberg-rust path as the long term
>>>>>> default for Comet/Iceberg scans, while it's still valuable to keep the
>>>>>> Iceberg Java integration as an option, especially for users who need
>>>>>> JVM-specific features. I'm hoping we can get consensus on a path to resolve
>>>>>> the shading issues (e.g. the approach proposed in
>>>>>> github.com/apache/iceberg/pull/13786) so the java option can move
>>>>>> forward.
>>>>>>
>>>>>> Thanks,
>>>>>> Huaxin
>>>>>>
>>>>>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <ye...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Andy for starting this thread!
>>>>>>>
>>>>>>> I've replied to the issue above but am posting here as well for
>>>>>>> better visibility:
>>>>>>>
>>>>>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the
>>>>>>> shading and circular dependency headaches of the Java approach. Easier
>>>>>>> maintenance typically lowers the entry barrier and encourages more
>>>>>>> community involvement, which is important for the project’s long-term
>>>>>>> health.
>>>>>>>
>>>>>>> It also makes sense to keep the Iceberg Java integration available
>>>>>>> as an experimental option for users who need its richer JVM-specific
>>>>>>> features while Rust becomes the long-term default. We don’t want to block
>>>>>>> contributions from anyone who truly needs the Java path.
>>>>>>>
>>>>>>> Best,
>>>>>>> Shawn
>>>>>>>
>>>>>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <an...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> There is a discussion in the DataFusion Comet community about the
>>>>>>>> future direction of our Iceberg support [1]. For example, should we
>>>>>>>> continue the efforts to integrate with the Iceberg Java project or focus
>>>>>>>> more on the iceberg-rust project.
>>>>>>>>
>>>>>>>> It would be great to also get input from the Iceberg community.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Andy.
>>>>>>>>
>>>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921
>>>>>>>>
>>>>>>>