You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@paimon.apache.org by Mike Dias via user <us...@paimon.apache.org> on 2026/02/16 05:45:00 UTC
Re: Paimon and Delta Lake compatibility
Btw, the Paimon Support in XTable PR got merged
https://github.com/apache/incubator-xtable/pull/742
Given that, would the Paimon community be interested in consolidating
Iceberg support in the XTable codebase rather than using the commit
callback approach?
On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <md...@atlassian.com> wrote:
> Hi Vinish and Nick, here is the draft PR ->
> https://github.com/apache/incubator-xtable/pull/742
>
> Looking forward to getting your feedback, especially on the Iceberg issue!
> :)
>
> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <md...@atlassian.com> wrote:
>
>> Hi Vinish, that's a good idea!
>>
>> I'll tidy up my branch and raise a draft pull request instead of an
>> issue; I think it will be easier to troubleshoot it by seeing the changes
>> and having a way to reproduce it.
>>
>> Thanks,
>> Mike Dias
>>
>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <vi...@apache.org> wrote:
>>
>>> Hi Mike.
>>>
>>> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg
>>> target issues you are seeing for the XTable Paimon source, can you create a
>>> GH issue? The XTable community can help in root causing the problem.
>>> https://github.com/apache/incubator-xtable/issues
>>>
>>> -Vinish
>>>
>>> On 2025/09/01 05:37:39 Mike Dias via user wrote:
>>> > Hi Nick, thanks for your reply!
>>> >
>>> > Yeah, I believe that creating the XTable source for Paimon is the best
>>> > approach: fewer hops, fewer dependencies, less maintenance overall.
>>> >
>>> > I have a prototype for the XTable Paimon source that works for Delta
>>> and
>>> > Hudi targets, but it does not work for the Iceberg target yet due to
>>> issues
>>> > with reading parquet fields with incorrect positions that I don't yet
>>> > understand why...
>>> >
>>> > Given that you have experience with Paimon and Iceberg, maybe I could
>>> get
>>> > some help troubleshooting that? That would be a huge help!
>>> >
>>> >
>>> >
>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <ni...@gmail.com>
>>> wrote:
>>> >
>>> > > Hi Mike,
>>> > >
>>> > > I help maintain Paimon's Iceberg Compatibility.
>>> > >
>>> > > The fastest way to accomplish your goal is the first option. It
>>> should not
>>> > > require code changes to Paimon or XTable. That said the feature has
>>> some
>>> > > tradeoffs.
>>> > >
>>> > > XTable is greatly preferred to adding Delta specific code to Paimon.
>>> I do
>>> > > think it would benefit Paimon adoption to be available in XTable [1].
>>> > >
>>> > > Paimon's Iceberg Compatibility works differently than XTable - it is
>>> a
>>> > > synchronous commit callback to translate metadata. This makes it
>>> really
>>> > > easy to enable and it simply supports latest snapshot queries. It
>>> does not
>>> > > yet support adding the full snapshot history (which I am working on
>>> > > https://github.com/apache/paimon/issues/6107) but still there will
>>> be
>>> > > some practical limitations around very long snapshot histories.
>>> > >
>>> > > Does the current feature set of Paimon-Iceberg compatibility support
>>> your
>>> > > use case?
>>> > > Would you be willing to contribute Paimon support in XTable?
>>> > >
>>> > > Nick
>>> > >
>>> > > [1] https://github.com/apache/incubator-xtable/issues/275
>>> > >
>>> > >
>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote:
>>> > > > Hello, we are currently evaluating using Paimon as our primary
>>> format for
>>> > > > landing data via Flink, but we still want to maintain
>>> compatibility with
>>> > > > Delta Lake, as we use Delta Sharing
>>> > > > <https://github.com/delta-io/delta-sharing> features extensively.
>>> > > >
>>> > > > Right now, it seems that there are three options to achieve that:
>>> > > >
>>> > > > - Enable the Paimon -> Iceberg
>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>
>>> > > Integration,
>>> > > > and then use XTable <https://xtable.apache.org/> to convert
>>> Iceberg
>>> > > > metadata to Delta Lake metadata.
>>> > > > - Implement a Paimon Source for XTable <
>>> https://xtable.apache.org/>
>>> > > and
>>> > > > use it to convert Delta Lake metadata directly from Paimon
>>> metadata
>>> > > > (read-optimize only).
>>> > > > - Implement a Commit Callback in Paimon that writes the Delta
>>> Lake
>>> > > > metadata directly, similar to what was done on the Iceberg
>>> integration
>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>.
>>> > > >
>>> > > > We are playing with the first option right now, but we would like
>>> to
>>> > > avoid
>>> > > > that going forward because the Iceberg conversion is unnecessary
>>> for us.
>>> > > So
>>> > > > the question is, would the Paimon community accept contributions to
>>> > > enable
>>> > > > Delta Lake compatibility, just like Iceberg's, or would the
>>> community
>>> > > > prefer to rely on XTable for such cross-table format conversions?
>>> > > >
>>> > > > Keen to hear your thoughts!
>>> > > >
>>> > > > Thanks
>>> > > > Mike Dias
>>> > > >
>>> >
>>> >
>>> >
>>> > --
>>> > Thanks,
>>> > Mike Dias
>>> >
>>>
>>
>
> --
> Thanks,
> Mike Dias
>
--
Thanks,
Mike Dias
Re: Paimon and Delta Lake compatibility
Posted by Mike Dias via user <us...@paimon.apache.org>.
Good point. There is an RFC for Deletion Vectors ->
https://github.com/apache/incubator-xtable/blob/main/rfc/rfc-2/2%20-%20Deletion%20Info%20Conversion.md,
but it hasn't been implemented yet.
Sounds like it is not something we could replace at this time. Maybe
sometime in the future.
On Thu, Feb 26, 2026 at 5:38 PM Jingsong Li <ji...@gmail.com> wrote:
> Hi Mike,
>
> You are right, but this requires a more detailed evaluation to ensure
> that the integration of the two projects will not affect existing
> users.
>
> Also, I am not sure if the ability to delete vectors is currently
> implemented in XTable.
>
> Best,
> Jingsong
>
> On Thu, Feb 26, 2026 at 2:34 PM Mike Dias <md...@atlassian.com> wrote:
> >
> > Thanks Jingsong. I believe it should be possible to hook the Xtable
> conversion as a CommitCallback too, ensuring the same real-time generation.
> >
> > Then, Paimon codebase doesn't need to worry about Iceberg logic,
> artifacts, and upgrades anymore, offloading all that to the Xtable codebase.
> >
> > On Thu, Feb 26, 2026 at 3:27 PM Jingsong Li <ji...@gmail.com>
> wrote:
> >>
> >> Hi Mike,
> >>
> >> The commit callback can ensure real-time generation of Iceberg
> >> snapshots. XTable is another implementation route, and they are
> >> developed in parallel.
> >>
> >> Best,
> >> Jingsong
> >>
> >> On Mon, Feb 16, 2026 at 1:45 PM Mike Dias via user
> >> <us...@paimon.apache.org> wrote:
> >> >
> >> > Btw, the Paimon Support in XTable PR got merged
> https://github.com/apache/incubator-xtable/pull/742
> >> >
> >> > Given that, would the Paimon community be interested in consolidating
> Iceberg support in the XTable codebase rather than using the commit
> callback approach?
> >> >
> >> > On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <md...@atlassian.com> wrote:
> >> >>
> >> >> Hi Vinish and Nick, here is the draft PR ->
> https://github.com/apache/incubator-xtable/pull/742
> >> >>
> >> >> Looking forward to getting your feedback, especially on the Iceberg
> issue! :)
> >> >>
> >> >> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <md...@atlassian.com>
> wrote:
> >> >>>
> >> >>> Hi Vinish, that's a good idea!
> >> >>>
> >> >>> I'll tidy up my branch and raise a draft pull request instead of an
> issue; I think it will be easier to troubleshoot it by seeing the changes
> and having a way to reproduce it.
> >> >>>
> >> >>> Thanks,
> >> >>> Mike Dias
> >> >>>
> >> >>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <vi...@apache.org>
> wrote:
> >> >>>>
> >> >>>> Hi Mike.
> >> >>>>
> >> >>>> I'm Vinish one of the PPMC members for XTable. Regarding the
> iceberg target issues you are seeing for the XTable Paimon source, can you
> create a GH issue? The XTable community can help in root causing the
> problem.
> >> >>>> https://github.com/apache/incubator-xtable/issues
> >> >>>>
> >> >>>> -Vinish
> >> >>>>
> >> >>>> On 2025/09/01 05:37:39 Mike Dias via user wrote:
> >> >>>> > Hi Nick, thanks for your reply!
> >> >>>> >
> >> >>>> > Yeah, I believe that creating the XTable source for Paimon is
> the best
> >> >>>> > approach: fewer hops, fewer dependencies, less maintenance
> overall.
> >> >>>> >
> >> >>>> > I have a prototype for the XTable Paimon source that works for
> Delta and
> >> >>>> > Hudi targets, but it does not work for the Iceberg target yet
> due to issues
> >> >>>> > with reading parquet fields with incorrect positions that I
> don't yet
> >> >>>> > understand why...
> >> >>>> >
> >> >>>> > Given that you have experience with Paimon and Iceberg, maybe I
> could get
> >> >>>> > some help troubleshooting that? That would be a huge help!
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <
> nickdelnano@gmail.com> wrote:
> >> >>>> >
> >> >>>> > > Hi Mike,
> >> >>>> > >
> >> >>>> > > I help maintain Paimon's Iceberg Compatibility.
> >> >>>> > >
> >> >>>> > > The fastest way to accomplish your goal is the first option.
> It should not
> >> >>>> > > require code changes to Paimon or XTable. That said the
> feature has some
> >> >>>> > > tradeoffs.
> >> >>>> > >
> >> >>>> > > XTable is greatly preferred to adding Delta specific code to
> Paimon. I do
> >> >>>> > > think it would benefit Paimon adoption to be available in
> XTable [1].
> >> >>>> > >
> >> >>>> > > Paimon's Iceberg Compatibility works differently than XTable -
> it is a
> >> >>>> > > synchronous commit callback to translate metadata. This makes
> it really
> >> >>>> > > easy to enable and it simply supports latest snapshot queries.
> It does not
> >> >>>> > > yet support adding the full snapshot history (which I am
> working on
> >> >>>> > > https://github.com/apache/paimon/issues/6107) but still there
> will be
> >> >>>> > > some practical limitations around very long snapshot histories.
> >> >>>> > >
> >> >>>> > > Does the current feature set of Paimon-Iceberg compatibility
> support your
> >> >>>> > > use case?
> >> >>>> > > Would you be willing to contribute Paimon support in XTable?
> >> >>>> > >
> >> >>>> > > Nick
> >> >>>> > >
> >> >>>> > > [1] https://github.com/apache/incubator-xtable/issues/275
> >> >>>> > >
> >> >>>> > >
> >> >>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote:
> >> >>>> > > > Hello, we are currently evaluating using Paimon as our
> primary format for
> >> >>>> > > > landing data via Flink, but we still want to maintain
> compatibility with
> >> >>>> > > > Delta Lake, as we use Delta Sharing
> >> >>>> > > > <https://github.com/delta-io/delta-sharing> features
> extensively.
> >> >>>> > > >
> >> >>>> > > > Right now, it seems that there are three options to achieve
> that:
> >> >>>> > > >
> >> >>>> > > > - Enable the Paimon -> Iceberg
> >> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>
> >> >>>> > > Integration,
> >> >>>> > > > and then use XTable <https://xtable.apache.org/> to
> convert Iceberg
> >> >>>> > > > metadata to Delta Lake metadata.
> >> >>>> > > > - Implement a Paimon Source for XTable <
> https://xtable.apache.org/>
> >> >>>> > > and
> >> >>>> > > > use it to convert Delta Lake metadata directly from
> Paimon metadata
> >> >>>> > > > (read-optimize only).
> >> >>>> > > > - Implement a Commit Callback in Paimon that writes the
> Delta Lake
> >> >>>> > > > metadata directly, similar to what was done on the
> Iceberg integration
> >> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/
> >.
> >> >>>> > > >
> >> >>>> > > > We are playing with the first option right now, but we would
> like to
> >> >>>> > > avoid
> >> >>>> > > > that going forward because the Iceberg conversion is
> unnecessary for us.
> >> >>>> > > So
> >> >>>> > > > the question is, would the Paimon community accept
> contributions to
> >> >>>> > > enable
> >> >>>> > > > Delta Lake compatibility, just like Iceberg's, or would the
> community
> >> >>>> > > > prefer to rely on XTable for such cross-table format
> conversions?
> >> >>>> > > >
> >> >>>> > > > Keen to hear your thoughts!
> >> >>>> > > >
> >> >>>> > > > Thanks
> >> >>>> > > > Mike Dias
> >> >>>> > > >
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > --
> >> >>>> > Thanks,
> >> >>>> > Mike Dias
> >> >>>> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Thanks,
> >> >> Mike Dias
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks,
> >> > Mike Dias
> >
> >
> >
> > --
> > Thanks,
> > Mike Dias
>
--
Thanks,
Mike Dias
Re: Paimon and Delta Lake compatibility
Posted by Jingsong Li <ji...@gmail.com>.
Hi Mike,
You are right, but this requires a more detailed evaluation to ensure
that the integration of the two projects will not affect existing
users.
Also, I am not sure if the ability to delete vectors is currently
implemented in XTable.
Best,
Jingsong
On Thu, Feb 26, 2026 at 2:34 PM Mike Dias <md...@atlassian.com> wrote:
>
> Thanks Jingsong. I believe it should be possible to hook the Xtable conversion as a CommitCallback too, ensuring the same real-time generation.
>
> Then, Paimon codebase doesn't need to worry about Iceberg logic, artifacts, and upgrades anymore, offloading all that to the Xtable codebase.
>
> On Thu, Feb 26, 2026 at 3:27 PM Jingsong Li <ji...@gmail.com> wrote:
>>
>> Hi Mike,
>>
>> The commit callback can ensure real-time generation of Iceberg
>> snapshots. XTable is another implementation route, and they are
>> developed in parallel.
>>
>> Best,
>> Jingsong
>>
>> On Mon, Feb 16, 2026 at 1:45 PM Mike Dias via user
>> <us...@paimon.apache.org> wrote:
>> >
>> > Btw, the Paimon Support in XTable PR got merged https://github.com/apache/incubator-xtable/pull/742
>> >
>> > Given that, would the Paimon community be interested in consolidating Iceberg support in the XTable codebase rather than using the commit callback approach?
>> >
>> > On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <md...@atlassian.com> wrote:
>> >>
>> >> Hi Vinish and Nick, here is the draft PR -> https://github.com/apache/incubator-xtable/pull/742
>> >>
>> >> Looking forward to getting your feedback, especially on the Iceberg issue! :)
>> >>
>> >> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <md...@atlassian.com> wrote:
>> >>>
>> >>> Hi Vinish, that's a good idea!
>> >>>
>> >>> I'll tidy up my branch and raise a draft pull request instead of an issue; I think it will be easier to troubleshoot it by seeing the changes and having a way to reproduce it.
>> >>>
>> >>> Thanks,
>> >>> Mike Dias
>> >>>
>> >>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <vi...@apache.org> wrote:
>> >>>>
>> >>>> Hi Mike.
>> >>>>
>> >>>> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg target issues you are seeing for the XTable Paimon source, can you create a GH issue? The XTable community can help in root causing the problem.
>> >>>> https://github.com/apache/incubator-xtable/issues
>> >>>>
>> >>>> -Vinish
>> >>>>
>> >>>> On 2025/09/01 05:37:39 Mike Dias via user wrote:
>> >>>> > Hi Nick, thanks for your reply!
>> >>>> >
>> >>>> > Yeah, I believe that creating the XTable source for Paimon is the best
>> >>>> > approach: fewer hops, fewer dependencies, less maintenance overall.
>> >>>> >
>> >>>> > I have a prototype for the XTable Paimon source that works for Delta and
>> >>>> > Hudi targets, but it does not work for the Iceberg target yet due to issues
>> >>>> > with reading parquet fields with incorrect positions that I don't yet
>> >>>> > understand why...
>> >>>> >
>> >>>> > Given that you have experience with Paimon and Iceberg, maybe I could get
>> >>>> > some help troubleshooting that? That would be a huge help!
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <ni...@gmail.com> wrote:
>> >>>> >
>> >>>> > > Hi Mike,
>> >>>> > >
>> >>>> > > I help maintain Paimon's Iceberg Compatibility.
>> >>>> > >
>> >>>> > > The fastest way to accomplish your goal is the first option. It should not
>> >>>> > > require code changes to Paimon or XTable. That said the feature has some
>> >>>> > > tradeoffs.
>> >>>> > >
>> >>>> > > XTable is greatly preferred to adding Delta specific code to Paimon. I do
>> >>>> > > think it would benefit Paimon adoption to be available in XTable [1].
>> >>>> > >
>> >>>> > > Paimon's Iceberg Compatibility works differently than XTable - it is a
>> >>>> > > synchronous commit callback to translate metadata. This makes it really
>> >>>> > > easy to enable and it simply supports latest snapshot queries. It does not
>> >>>> > > yet support adding the full snapshot history (which I am working on
>> >>>> > > https://github.com/apache/paimon/issues/6107) but still there will be
>> >>>> > > some practical limitations around very long snapshot histories.
>> >>>> > >
>> >>>> > > Does the current feature set of Paimon-Iceberg compatibility support your
>> >>>> > > use case?
>> >>>> > > Would you be willing to contribute Paimon support in XTable?
>> >>>> > >
>> >>>> > > Nick
>> >>>> > >
>> >>>> > > [1] https://github.com/apache/incubator-xtable/issues/275
>> >>>> > >
>> >>>> > >
>> >>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote:
>> >>>> > > > Hello, we are currently evaluating using Paimon as our primary format for
>> >>>> > > > landing data via Flink, but we still want to maintain compatibility with
>> >>>> > > > Delta Lake, as we use Delta Sharing
>> >>>> > > > <https://github.com/delta-io/delta-sharing> features extensively.
>> >>>> > > >
>> >>>> > > > Right now, it seems that there are three options to achieve that:
>> >>>> > > >
>> >>>> > > > - Enable the Paimon -> Iceberg
>> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>
>> >>>> > > Integration,
>> >>>> > > > and then use XTable <https://xtable.apache.org/> to convert Iceberg
>> >>>> > > > metadata to Delta Lake metadata.
>> >>>> > > > - Implement a Paimon Source for XTable <https://xtable.apache.org/>
>> >>>> > > and
>> >>>> > > > use it to convert Delta Lake metadata directly from Paimon metadata
>> >>>> > > > (read-optimize only).
>> >>>> > > > - Implement a Commit Callback in Paimon that writes the Delta Lake
>> >>>> > > > metadata directly, similar to what was done on the Iceberg integration
>> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>.
>> >>>> > > >
>> >>>> > > > We are playing with the first option right now, but we would like to
>> >>>> > > avoid
>> >>>> > > > that going forward because the Iceberg conversion is unnecessary for us.
>> >>>> > > So
>> >>>> > > > the question is, would the Paimon community accept contributions to
>> >>>> > > enable
>> >>>> > > > Delta Lake compatibility, just like Iceberg's, or would the community
>> >>>> > > > prefer to rely on XTable for such cross-table format conversions?
>> >>>> > > >
>> >>>> > > > Keen to hear your thoughts!
>> >>>> > > >
>> >>>> > > > Thanks
>> >>>> > > > Mike Dias
>> >>>> > > >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > Thanks,
>> >>>> > Mike Dias
>> >>>> >
>> >>
>> >>
>> >>
>> >> --
>> >> Thanks,
>> >> Mike Dias
>> >
>> >
>> >
>> > --
>> > Thanks,
>> > Mike Dias
>
>
>
> --
> Thanks,
> Mike Dias
Re: Paimon and Delta Lake compatibility
Posted by Mike Dias via user <us...@paimon.apache.org>.
Thanks Jingsong. I believe it should be possible to hook the Xtable
conversion as a CommitCallback too, ensuring the same real-time generation.
Then, Paimon codebase doesn't need to worry about Iceberg logic, artifacts,
and upgrades anymore, offloading all that to the Xtable codebase.
On Thu, Feb 26, 2026 at 3:27 PM Jingsong Li <ji...@gmail.com> wrote:
> Hi Mike,
>
> The commit callback can ensure real-time generation of Iceberg
> snapshots. XTable is another implementation route, and they are
> developed in parallel.
>
> Best,
> Jingsong
>
> On Mon, Feb 16, 2026 at 1:45 PM Mike Dias via user
> <us...@paimon.apache.org> wrote:
> >
> > Btw, the Paimon Support in XTable PR got merged
> https://github.com/apache/incubator-xtable/pull/742
> >
> > Given that, would the Paimon community be interested in consolidating
> Iceberg support in the XTable codebase rather than using the commit
> callback approach?
> >
> > On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <md...@atlassian.com> wrote:
> >>
> >> Hi Vinish and Nick, here is the draft PR ->
> https://github.com/apache/incubator-xtable/pull/742
> >>
> >> Looking forward to getting your feedback, especially on the Iceberg
> issue! :)
> >>
> >> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <md...@atlassian.com> wrote:
> >>>
> >>> Hi Vinish, that's a good idea!
> >>>
> >>> I'll tidy up my branch and raise a draft pull request instead of an
> issue; I think it will be easier to troubleshoot it by seeing the changes
> and having a way to reproduce it.
> >>>
> >>> Thanks,
> >>> Mike Dias
> >>>
> >>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <vi...@apache.org> wrote:
> >>>>
> >>>> Hi Mike.
> >>>>
> >>>> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg
> target issues you are seeing for the XTable Paimon source, can you create a
> GH issue? The XTable community can help in root causing the problem.
> >>>> https://github.com/apache/incubator-xtable/issues
> >>>>
> >>>> -Vinish
> >>>>
> >>>> On 2025/09/01 05:37:39 Mike Dias via user wrote:
> >>>> > Hi Nick, thanks for your reply!
> >>>> >
> >>>> > Yeah, I believe that creating the XTable source for Paimon is the
> best
> >>>> > approach: fewer hops, fewer dependencies, less maintenance overall.
> >>>> >
> >>>> > I have a prototype for the XTable Paimon source that works for
> Delta and
> >>>> > Hudi targets, but it does not work for the Iceberg target yet due
> to issues
> >>>> > with reading parquet fields with incorrect positions that I don't
> yet
> >>>> > understand why...
> >>>> >
> >>>> > Given that you have experience with Paimon and Iceberg, maybe I
> could get
> >>>> > some help troubleshooting that? That would be a huge help!
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <
> nickdelnano@gmail.com> wrote:
> >>>> >
> >>>> > > Hi Mike,
> >>>> > >
> >>>> > > I help maintain Paimon's Iceberg Compatibility.
> >>>> > >
> >>>> > > The fastest way to accomplish your goal is the first option. It
> should not
> >>>> > > require code changes to Paimon or XTable. That said the feature
> has some
> >>>> > > tradeoffs.
> >>>> > >
> >>>> > > XTable is greatly preferred to adding Delta specific code to
> Paimon. I do
> >>>> > > think it would benefit Paimon adoption to be available in XTable
> [1].
> >>>> > >
> >>>> > > Paimon's Iceberg Compatibility works differently than XTable - it
> is a
> >>>> > > synchronous commit callback to translate metadata. This makes it
> really
> >>>> > > easy to enable and it simply supports latest snapshot queries. It
> does not
> >>>> > > yet support adding the full snapshot history (which I am working
> on
> >>>> > > https://github.com/apache/paimon/issues/6107) but still there
> will be
> >>>> > > some practical limitations around very long snapshot histories.
> >>>> > >
> >>>> > > Does the current feature set of Paimon-Iceberg compatibility
> support your
> >>>> > > use case?
> >>>> > > Would you be willing to contribute Paimon support in XTable?
> >>>> > >
> >>>> > > Nick
> >>>> > >
> >>>> > > [1] https://github.com/apache/incubator-xtable/issues/275
> >>>> > >
> >>>> > >
> >>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote:
> >>>> > > > Hello, we are currently evaluating using Paimon as our primary
> format for
> >>>> > > > landing data via Flink, but we still want to maintain
> compatibility with
> >>>> > > > Delta Lake, as we use Delta Sharing
> >>>> > > > <https://github.com/delta-io/delta-sharing> features
> extensively.
> >>>> > > >
> >>>> > > > Right now, it seems that there are three options to achieve
> that:
> >>>> > > >
> >>>> > > > - Enable the Paimon -> Iceberg
> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>
> >>>> > > Integration,
> >>>> > > > and then use XTable <https://xtable.apache.org/> to convert
> Iceberg
> >>>> > > > metadata to Delta Lake metadata.
> >>>> > > > - Implement a Paimon Source for XTable <
> https://xtable.apache.org/>
> >>>> > > and
> >>>> > > > use it to convert Delta Lake metadata directly from Paimon
> metadata
> >>>> > > > (read-optimize only).
> >>>> > > > - Implement a Commit Callback in Paimon that writes the
> Delta Lake
> >>>> > > > metadata directly, similar to what was done on the Iceberg
> integration
> >>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>.
> >>>> > > >
> >>>> > > > We are playing with the first option right now, but we would
> like to
> >>>> > > avoid
> >>>> > > > that going forward because the Iceberg conversion is
> unnecessary for us.
> >>>> > > So
> >>>> > > > the question is, would the Paimon community accept
> contributions to
> >>>> > > enable
> >>>> > > > Delta Lake compatibility, just like Iceberg's, or would the
> community
> >>>> > > > prefer to rely on XTable for such cross-table format
> conversions?
> >>>> > > >
> >>>> > > > Keen to hear your thoughts!
> >>>> > > >
> >>>> > > > Thanks
> >>>> > > > Mike Dias
> >>>> > > >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > Thanks,
> >>>> > Mike Dias
> >>>> >
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Mike Dias
> >
> >
> >
> > --
> > Thanks,
> > Mike Dias
>
--
Thanks,
Mike Dias
Re: Paimon and Delta Lake compatibility
Posted by Jingsong Li <ji...@gmail.com>.
Hi Mike,
The commit callback can ensure real-time generation of Iceberg
snapshots. XTable is another implementation route, and they are
developed in parallel.
Best,
Jingsong
On Mon, Feb 16, 2026 at 1:45 PM Mike Dias via user
<us...@paimon.apache.org> wrote:
>
> Btw, the Paimon Support in XTable PR got merged https://github.com/apache/incubator-xtable/pull/742
>
> Given that, would the Paimon community be interested in consolidating Iceberg support in the XTable codebase rather than using the commit callback approach?
>
> On Mon, Sep 8, 2025 at 3:55 PM Mike Dias <md...@atlassian.com> wrote:
>>
>> Hi Vinish and Nick, here is the draft PR -> https://github.com/apache/incubator-xtable/pull/742
>>
>> Looking forward to getting your feedback, especially on the Iceberg issue! :)
>>
>> On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <md...@atlassian.com> wrote:
>>>
>>> Hi Vinish, that's a good idea!
>>>
>>> I'll tidy up my branch and raise a draft pull request instead of an issue; I think it will be easier to troubleshoot it by seeing the changes and having a way to reproduce it.
>>>
>>> Thanks,
>>> Mike Dias
>>>
>>> On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <vi...@apache.org> wrote:
>>>>
>>>> Hi Mike.
>>>>
>>>> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg target issues you are seeing for the XTable Paimon source, can you create a GH issue? The XTable community can help in root causing the problem.
>>>> https://github.com/apache/incubator-xtable/issues
>>>>
>>>> -Vinish
>>>>
>>>> On 2025/09/01 05:37:39 Mike Dias via user wrote:
>>>> > Hi Nick, thanks for your reply!
>>>> >
>>>> > Yeah, I believe that creating the XTable source for Paimon is the best
>>>> > approach: fewer hops, fewer dependencies, less maintenance overall.
>>>> >
>>>> > I have a prototype for the XTable Paimon source that works for Delta and
>>>> > Hudi targets, but it does not work for the Iceberg target yet due to issues
>>>> > with reading parquet fields with incorrect positions that I don't yet
>>>> > understand why...
>>>> >
>>>> > Given that you have experience with Paimon and Iceberg, maybe I could get
>>>> > some help troubleshooting that? That would be a huge help!
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <ni...@gmail.com> wrote:
>>>> >
>>>> > > Hi Mike,
>>>> > >
>>>> > > I help maintain Paimon's Iceberg Compatibility.
>>>> > >
>>>> > > The fastest way to accomplish your goal is the first option. It should not
>>>> > > require code changes to Paimon or XTable. That said the feature has some
>>>> > > tradeoffs.
>>>> > >
>>>> > > XTable is greatly preferred to adding Delta specific code to Paimon. I do
>>>> > > think it would benefit Paimon adoption to be available in XTable [1].
>>>> > >
>>>> > > Paimon's Iceberg Compatibility works differently than XTable - it is a
>>>> > > synchronous commit callback to translate metadata. This makes it really
>>>> > > easy to enable and it simply supports latest snapshot queries. It does not
>>>> > > yet support adding the full snapshot history (which I am working on
>>>> > > https://github.com/apache/paimon/issues/6107) but still there will be
>>>> > > some practical limitations around very long snapshot histories.
>>>> > >
>>>> > > Does the current feature set of Paimon-Iceberg compatibility support your
>>>> > > use case?
>>>> > > Would you be willing to contribute Paimon support in XTable?
>>>> > >
>>>> > > Nick
>>>> > >
>>>> > > [1] https://github.com/apache/incubator-xtable/issues/275
>>>> > >
>>>> > >
>>>> > > On 2025/08/08 05:44:03 Mike Dias via user wrote:
>>>> > > > Hello, we are currently evaluating using Paimon as our primary format for
>>>> > > > landing data via Flink, but we still want to maintain compatibility with
>>>> > > > Delta Lake, as we use Delta Sharing
>>>> > > > <https://github.com/delta-io/delta-sharing> features extensively.
>>>> > > >
>>>> > > > Right now, it seems that there are three options to achieve that:
>>>> > > >
>>>> > > > - Enable the Paimon -> Iceberg
>>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>
>>>> > > Integration,
>>>> > > > and then use XTable <https://xtable.apache.org/> to convert Iceberg
>>>> > > > metadata to Delta Lake metadata.
>>>> > > > - Implement a Paimon Source for XTable <https://xtable.apache.org/>
>>>> > > and
>>>> > > > use it to convert Delta Lake metadata directly from Paimon metadata
>>>> > > > (read-optimize only).
>>>> > > > - Implement a Commit Callback in Paimon that writes the Delta Lake
>>>> > > > metadata directly, similar to what was done on the Iceberg integration
>>>> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>.
>>>> > > >
>>>> > > > We are playing with the first option right now, but we would like to
>>>> > > avoid
>>>> > > > that going forward because the Iceberg conversion is unnecessary for us.
>>>> > > So
>>>> > > > the question is, would the Paimon community accept contributions to
>>>> > > enable
>>>> > > > Delta Lake compatibility, just like Iceberg's, or would the community
>>>> > > > prefer to rely on XTable for such cross-table format conversions?
>>>> > > >
>>>> > > > Keen to hear your thoughts!
>>>> > > >
>>>> > > > Thanks
>>>> > > > Mike Dias
>>>> > > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Thanks,
>>>> > Mike Dias
>>>> >
>>
>>
>>
>> --
>> Thanks,
>> Mike Dias
>
>
>
> --
> Thanks,
> Mike Dias