You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@xtable.apache.org by Vinish Reddy Pannala <vi...@gmail.com> on 2026/07/14 01:09:06 UTC
[DISCUSS] xtable-java-runtime: a Spark-free runtime enabled by Delta Kernel
Hi all,
I'd like to open a discussion on publishing an xtable-java-runtime - a
lightweight, Spark-free runtime for running XTable metadata sync - now that
Delta Kernel support has landed on main.
Motivation
Until now, any XTable sync involving Delta effectively required a
SparkSession, since the Delta source/target went through delta-spark's
DeltaLog APIs. The Hudi and Iceberg paths are already pure Java, so Delta
was the one format anchoring us to Spark. Today the runtime options we ship
(the xtable-utilities CLI and xtable-spark-runtime, #843) both carry Spark.
With Delta Kernel PR's now merged, Delta can be read and written without
Spark. That removes the last hard Spark dependency in the sync path, which
means we can publish a genuinely Spark-free, pure-Java runtime jar covering
Hudi, Iceberg, and Delta (via Kernel).
Why it matters - non-Spark engines
A Java-only runtime makes XTable embeddable anywhere on the JVM without
dragging in Spark:
- Flink jobs syncing metadata in-line
- Trino / Presto plugins and connectors
- Plain Java services, functions, or orchestration steps
This drastically shrinks the footprint and the dependency-conflict surface
versus a Spark bundle, and opens XTable to the large set of users who
aren't on Spark.
Context
We discussed this during the community sync this morning and agreed it's
worth pursuing - the notes capture that with Delta Kernel merged, an
xtable-java-runtime jar can be published with Delta Kernel as
source/target. Full notes:
https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing
Questions for the list
- Do we agree an xtable-java-runtime is worth publishing in the next
release?
- Any concerns about Delta Kernel feature parity for a Kernel-only runtime
(e.g. deletion vectors, issue #713)?
Looking forward to your thoughts.
Thanks,
Vinish
Re: [DISCUSS] xtable-java-runtime: a Spark-free runtime enabled by Delta Kernel
Posted by Vinish <vi...@gmail.com>.
Thanks Tim. I left review comments on the RFC in PR #612.They are mostly
questions rather than blockers, and I am +1 on the direction.
On Thu, Aug 13, 2026 05:16 AM, Tim Brown <ti...@gmail.com> wrote:
> I will revive the changes in PR #618. I plan on breaking down the full set
> of the changes to first get the test framework setup and merged, then I
> will break out the individual modules.
> It would be good to have some initial review on the RFC before we go much
> further to make sure things are aligned
> https://github.com/apache/incubator-xtable/pull/612
>
> On Mon, Aug 10, 2026 at 7:34 PM Vinish <vi...@gmail.com>
> wrote:
>
> > Hi Tim,
> >
> > I am withdrawing the xtable-java-runtime proposal. A new module needs
> > application-specific code, and this one does not have any.
> >
> > Do you want to revive PR #618, or should someone pick it up against the
> > current main? It predates Paimon, Parquet and Kernel.
> >
> > On your point about Delta Kernel, I have filed #886 for the
> > ITConversionController gap.
> > https://github.com/apache/incubator-xtable/issues/886.
> > We will pick it up regardless of how the packaging question lands, since
> it
> > is the precondition for making Kernel the default Delta path.
> >
> > Thanks,
> > Vinish
> >
> > On Sun, Jul 19, 2026 08:10 AM, Tim Brown <ti...@gmail.com> wrote:
> >
> > > If the goal is to make it easier for java applications, I think we
> should
> > > just go with the proposal in RFC-2. I have a now outdated
> implementation
> > > [1] of this including a bundle validation flow. The proposed jar is
> > > excluding Paimon for some reason and I think we should aim to support
> all
> > > the table formats that are supported. Bundling all of this will just
> lead
> > > us back to an overly large jar so it is key to keep things modular.
> > >
> > > Generally when people are building java applications, they are building
> > > bundled jars when they want to run these applications so it doesn't
> seem
> > > like a bundled jar is really the requirement here since they will be
> > > bundling the required dependencies declared by our jar's manifest. The
> > need
> > > here is to make sure the manifest is as lean as possible so someone
> that
> > > just wants delta-kernel and Iceberg will not pull in Hudi, Paimon, or
> any
> > > spark dependency.
> > >
> > > If there is a secondary concern that the user will already have their
> own
> > > version of some of the required dependencies that may conflict with
> what
> > we
> > > declare, then it makes sense to also publish our own bundled jars with
> > > relocated dependencies for these lean modules.
> > >
> > > The only reason I can see for adding yet another module to the project
> is
> > > if there is some application specific code that needs to be added. This
> > is
> > > the case for the spark proposal since it will include pre-built
> solutions
> > > for how to integrate XTable into existing writers.
> > >
> > >
> > > 1.
> > >
> > >
> >
> https://github.com/apache/incubator-xtable/pull/618/changes#diff-b818b66a60f0848d9b846b5f966ca959be112f27a12694fa9f79002d9ee9e07e
> > >
> > > -Tim
> > >
> > > On Fri, Jul 17, 2026 at 12:58 PM Vaibhav Kumar <kmickey.1992@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Tim,
> > > >
> > > >
> > > >
> > > > Thanks for the response and sharing the RFC.
> > > >
> > > > You're right that we need to be clear on the goal. Let me be precise
> as
> > > per
> > > > our discussion in the last sync call : the goal is not to replace or
> > > patch
> > > > the utilities jar. It is to allow XTable to be used as a drop-in
> > library
> > > in
> > > > JVM applications that do not run Spark at all — Flink jobs,
> > Trino/Presto
> > > > plugins, plain Java services. The utility jar cannot serve that goal
> > > even
> > > > if we slimmed it down, because it is designed as a CLI tool and
> > bundles
> > > > Spark by design.
> > > >
> > > >
> > > > This actually lines up directly with the pain points your RFC-2 [1]
> > > > identifies:
> > > >
> > > >
> > > > ▎ "When running the prebuilt utilities bundle jar, the user will
> > bring
> > > in
> > > > all the required
> > > > ▎ dependencies which currently include three table formats, spark,
> > and
> > > > hadoop. This results in a
> > > > ▎ very large jar containing more than the user really needs."
> > > >
> > > >
> > > >
> > > > ▎ "When building your own jar, you can run into issues with
> conflicts
> > > in
> > > > versions for the
> > > > ▎ specific table formats you want to use since the user will often
> > > > already have an implementation
> > > > ▎ of at least one format on their classpath."
> > > >
> > > > An xtable-java-runtime addresses both pain points directly for the
> > > > non-Spark user segment. It would be a shaded jar with relocated
> > > > dependencies — the same pattern used in xtable-spark-runtime(#843) —
> > > > bundling Delta Kernel, Hudi Java client, and Iceberg core with no
> > Spark.
> > > > Because dependencies are relocated inside the jar, version conflicts
> on
> > > the
> > > > user's classpath are avoided. This is exactly the per-format shaded
> jar
> > > > approach your RFC-2 proposes.
> > > >
> > > >
> > > > To your point on jar structure: it would be a shaded fat jar with
> > > relocated
> > > > dependencies. Users would consume it via --jars or as a Maven
> > dependency,
> > > > and the dependency-reduced POM would expose no transitive engine
> > > > dependencies.
> > > >
> > > >
> > > > On Delta Kernel readiness — you are right that it is not yet
> integrated
> > > > into ITConversionController as a first-class path alongside the
> > > Spark-based
> > > > Delta tests. The dedicated test suite (ITDeltaKernelConversionSource,
> > > > TestDeltaKernelReadWriteIntegration, TestDeltaKernelSync) covers the
> > > > Kernel path in depth, but I agree the shared harness is the right
> bar.
> > > > Would it make sense to track the ITConversionController integration
> as
> > a
> > > > prerequisite for the java-runtime publish, so we can keep the
> > discussion
> > > > moving forward while that work gets done in parallel?
> > > >
> > > >
> > > > To summarise — the xtable-java-runtime is not a workaround for the
> > module
> > > > restructuring RFC. It is one concrete deliverable that RFC-2 leads
> to,
> > > > scoped narrowly to a packaging module so it does not depend on the
> full
> > > > restructure being complete first.
> > > >
> > > >
> > > > Happy to open a tracking issue for the ITConversionController gap and
> > put
> > > > together a more detailed design doc if the community agrees with the
> > > > direction.
> > > >
> > > > Thanks,
> > > >
> > > > Vaibhav
> > > >
> > > >
> > > >
> > > > On Tue, Jul 14, 2026 at 5:53 PM Tim Brown <ti...@gmail.com>
> > > wrote:
> > > >
> > > > > I think that we need to figure out what the goal is before
> proposing
> > > the
> > > > > structure. If the goal is to allow a standalone java jar to run
> > > > > conversions, I would suggest we simply revisit the utilities jar or
> > the
> > > > web
> > > > > service for this.
> > > > >
> > > > > The main issue with the utilities jar was that it was bundling all
> > > sorts
> > > > of
> > > > > things, including spark. We have a similar issue with the layout of
> > the
> > > > > project in general where it is hard to pick and choose the lean set
> > of
> > > > > dependencies that you want for your particular use case. That was
> the
> > > > > reason for the RFC for module restructuring [1].
> > > > >
> > > > > Before starting any work on this there are a few questions that
> need
> > to
> > > > be
> > > > > answered. What will the xtable-java-runtime jar look like? Will it
> > be a
> > > > > bundled jar? Will it have relocated dependencies or will it require
> > > users
> > > > > to provide the implementation of the table formats?
> > > > >
> > > > > Before declaring the Delta Kernel ready for use, I would like to
> see
> > it
> > > > > integrated into our integration test suite in
> ITConversionController
> > > [2].
> > > > > It is also worthwhile to look back at the recent issues reported
> for
> > > > Delta
> > > > > conversion and ensure they are also fixed in the Delta Kernel path.
> > > > >
> > > > > 1. https://github.com/apache/incubator-xtable/pull/612
> > > > > 2.
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java
> > > > >
> > > > > On Mon, Jul 13, 2026 at 10:31 PM Vaibhav Kumar <
> > kmickey.1992@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Vinish
> > > > > >
> > > > > > +1 for the idea, I can take this up.
> > > > > >
> > > > > > On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala <
> > > > > > vinishreddypannala@gmail.com> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to open a discussion on publishing an
> > xtable-java-runtime
> > > -
> > > > a
> > > > > > > lightweight, Spark-free runtime for running XTable metadata
> sync
> > -
> > > > now
> > > > > > that
> > > > > > > Delta Kernel support has landed on main.
> > > > > > >
> > > > > > > Motivation
> > > > > > >
> > > > > > > Until now, any XTable sync involving Delta effectively
> required a
> > > > > > > SparkSession, since the Delta source/target went through
> > > > delta-spark's
> > > > > > > DeltaLog APIs. The Hudi and Iceberg paths are already pure
> Java,
> > so
> > > > > Delta
> > > > > > > was the one format anchoring us to Spark. Today the runtime
> > options
> > > > we
> > > > > > ship
> > > > > > > (the xtable-utilities CLI and xtable-spark-runtime, #843) both
> > > carry
> > > > > > Spark.
> > > > > > >
> > > > > > > With Delta Kernel PR's now merged, Delta can be read and
> written
> > > > > without
> > > > > > > Spark. That removes the last hard Spark dependency in the sync
> > > path,
> > > > > > which
> > > > > > > means we can publish a genuinely Spark-free, pure-Java runtime
> > jar
> > > > > > covering
> > > > > > > Hudi, Iceberg, and Delta (via Kernel).
> > > > > > >
> > > > > > > Why it matters - non-Spark engines
> > > > > > >
> > > > > > > A Java-only runtime makes XTable embeddable anywhere on the JVM
> > > > without
> > > > > > > dragging in Spark:
> > > > > > >
> > > > > > > - Flink jobs syncing metadata in-line
> > > > > > > - Trino / Presto plugins and connectors
> > > > > > > - Plain Java services, functions, or orchestration steps
> > > > > > >
> > > > > > > This drastically shrinks the footprint and the
> > dependency-conflict
> > > > > > surface
> > > > > > > versus a Spark bundle, and opens XTable to the large set of
> users
> > > who
> > > > > > > aren't on Spark.
> > > > > > >
> > > > > > > Context
> > > > > > >
> > > > > > > We discussed this during the community sync this morning and
> > agreed
> > > > > it's
> > > > > > > worth pursuing - the notes capture that with Delta Kernel
> merged,
> > > an
> > > > > > > xtable-java-runtime jar can be published with Delta Kernel as
> > > > > > > source/target. Full notes:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing
> > > > > > >
> > > > > > > Questions for the list
> > > > > > >
> > > > > > > - Do we agree an xtable-java-runtime is worth publishing in the
> > > next
> > > > > > > release?
> > > > > > > - Any concerns about Delta Kernel feature parity for a
> > Kernel-only
> > > > > > runtime
> > > > > > > (e.g. deletion vectors, issue #713)?
> > > > > > >
> > > > > > > Looking forward to your thoughts.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Vinish
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] xtable-java-runtime: a Spark-free runtime enabled by Delta Kernel
Posted by Tim Brown <ti...@gmail.com>.
I will revive the changes in PR #618. I plan on breaking down the full set
of the changes to first get the test framework setup and merged, then I
will break out the individual modules.
It would be good to have some initial review on the RFC before we go much
further to make sure things are aligned
https://github.com/apache/incubator-xtable/pull/612
On Mon, Aug 10, 2026 at 7:34 PM Vinish <vi...@gmail.com> wrote:
> Hi Tim,
>
> I am withdrawing the xtable-java-runtime proposal. A new module needs
> application-specific code, and this one does not have any.
>
> Do you want to revive PR #618, or should someone pick it up against the
> current main? It predates Paimon, Parquet and Kernel.
>
> On your point about Delta Kernel, I have filed #886 for the
> ITConversionController gap.
> https://github.com/apache/incubator-xtable/issues/886.
> We will pick it up regardless of how the packaging question lands, since it
> is the precondition for making Kernel the default Delta path.
>
> Thanks,
> Vinish
>
> On Sun, Jul 19, 2026 08:10 AM, Tim Brown <ti...@gmail.com> wrote:
>
> > If the goal is to make it easier for java applications, I think we should
> > just go with the proposal in RFC-2. I have a now outdated implementation
> > [1] of this including a bundle validation flow. The proposed jar is
> > excluding Paimon for some reason and I think we should aim to support all
> > the table formats that are supported. Bundling all of this will just lead
> > us back to an overly large jar so it is key to keep things modular.
> >
> > Generally when people are building java applications, they are building
> > bundled jars when they want to run these applications so it doesn't seem
> > like a bundled jar is really the requirement here since they will be
> > bundling the required dependencies declared by our jar's manifest. The
> need
> > here is to make sure the manifest is as lean as possible so someone that
> > just wants delta-kernel and Iceberg will not pull in Hudi, Paimon, or any
> > spark dependency.
> >
> > If there is a secondary concern that the user will already have their own
> > version of some of the required dependencies that may conflict with what
> we
> > declare, then it makes sense to also publish our own bundled jars with
> > relocated dependencies for these lean modules.
> >
> > The only reason I can see for adding yet another module to the project is
> > if there is some application specific code that needs to be added. This
> is
> > the case for the spark proposal since it will include pre-built solutions
> > for how to integrate XTable into existing writers.
> >
> >
> > 1.
> >
> >
> https://github.com/apache/incubator-xtable/pull/618/changes#diff-b818b66a60f0848d9b846b5f966ca959be112f27a12694fa9f79002d9ee9e07e
> >
> > -Tim
> >
> > On Fri, Jul 17, 2026 at 12:58 PM Vaibhav Kumar <km...@gmail.com>
> > wrote:
> >
> > > Hi Tim,
> > >
> > >
> > >
> > > Thanks for the response and sharing the RFC.
> > >
> > > You're right that we need to be clear on the goal. Let me be precise as
> > per
> > > our discussion in the last sync call : the goal is not to replace or
> > patch
> > > the utilities jar. It is to allow XTable to be used as a drop-in
> library
> > in
> > > JVM applications that do not run Spark at all — Flink jobs,
> Trino/Presto
> > > plugins, plain Java services. The utility jar cannot serve that goal
> > even
> > > if we slimmed it down, because it is designed as a CLI tool and
> bundles
> > > Spark by design.
> > >
> > >
> > > This actually lines up directly with the pain points your RFC-2 [1]
> > > identifies:
> > >
> > >
> > > ▎ "When running the prebuilt utilities bundle jar, the user will
> bring
> > in
> > > all the required
> > > ▎ dependencies which currently include three table formats, spark,
> and
> > > hadoop. This results in a
> > > ▎ very large jar containing more than the user really needs."
> > >
> > >
> > >
> > > ▎ "When building your own jar, you can run into issues with conflicts
> > in
> > > versions for the
> > > ▎ specific table formats you want to use since the user will often
> > > already have an implementation
> > > ▎ of at least one format on their classpath."
> > >
> > > An xtable-java-runtime addresses both pain points directly for the
> > > non-Spark user segment. It would be a shaded jar with relocated
> > > dependencies — the same pattern used in xtable-spark-runtime(#843) —
> > > bundling Delta Kernel, Hudi Java client, and Iceberg core with no
> Spark.
> > > Because dependencies are relocated inside the jar, version conflicts on
> > the
> > > user's classpath are avoided. This is exactly the per-format shaded jar
> > > approach your RFC-2 proposes.
> > >
> > >
> > > To your point on jar structure: it would be a shaded fat jar with
> > relocated
> > > dependencies. Users would consume it via --jars or as a Maven
> dependency,
> > > and the dependency-reduced POM would expose no transitive engine
> > > dependencies.
> > >
> > >
> > > On Delta Kernel readiness — you are right that it is not yet integrated
> > > into ITConversionController as a first-class path alongside the
> > Spark-based
> > > Delta tests. The dedicated test suite (ITDeltaKernelConversionSource,
> > > TestDeltaKernelReadWriteIntegration, TestDeltaKernelSync) covers the
> > > Kernel path in depth, but I agree the shared harness is the right bar.
> > > Would it make sense to track the ITConversionController integration as
> a
> > > prerequisite for the java-runtime publish, so we can keep the
> discussion
> > > moving forward while that work gets done in parallel?
> > >
> > >
> > > To summarise — the xtable-java-runtime is not a workaround for the
> module
> > > restructuring RFC. It is one concrete deliverable that RFC-2 leads to,
> > > scoped narrowly to a packaging module so it does not depend on the full
> > > restructure being complete first.
> > >
> > >
> > > Happy to open a tracking issue for the ITConversionController gap and
> put
> > > together a more detailed design doc if the community agrees with the
> > > direction.
> > >
> > > Thanks,
> > >
> > > Vaibhav
> > >
> > >
> > >
> > > On Tue, Jul 14, 2026 at 5:53 PM Tim Brown <ti...@gmail.com>
> > wrote:
> > >
> > > > I think that we need to figure out what the goal is before proposing
> > the
> > > > structure. If the goal is to allow a standalone java jar to run
> > > > conversions, I would suggest we simply revisit the utilities jar or
> the
> > > web
> > > > service for this.
> > > >
> > > > The main issue with the utilities jar was that it was bundling all
> > sorts
> > > of
> > > > things, including spark. We have a similar issue with the layout of
> the
> > > > project in general where it is hard to pick and choose the lean set
> of
> > > > dependencies that you want for your particular use case. That was the
> > > > reason for the RFC for module restructuring [1].
> > > >
> > > > Before starting any work on this there are a few questions that need
> to
> > > be
> > > > answered. What will the xtable-java-runtime jar look like? Will it
> be a
> > > > bundled jar? Will it have relocated dependencies or will it require
> > users
> > > > to provide the implementation of the table formats?
> > > >
> > > > Before declaring the Delta Kernel ready for use, I would like to see
> it
> > > > integrated into our integration test suite in ITConversionController
> > [2].
> > > > It is also worthwhile to look back at the recent issues reported for
> > > Delta
> > > > conversion and ensure they are also fixed in the Delta Kernel path.
> > > >
> > > > 1. https://github.com/apache/incubator-xtable/pull/612
> > > > 2.
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java
> > > >
> > > > On Mon, Jul 13, 2026 at 10:31 PM Vaibhav Kumar <
> kmickey.1992@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hey Vinish
> > > > >
> > > > > +1 for the idea, I can take this up.
> > > > >
> > > > > On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala <
> > > > > vinishreddypannala@gmail.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to open a discussion on publishing an
> xtable-java-runtime
> > -
> > > a
> > > > > > lightweight, Spark-free runtime for running XTable metadata sync
> -
> > > now
> > > > > that
> > > > > > Delta Kernel support has landed on main.
> > > > > >
> > > > > > Motivation
> > > > > >
> > > > > > Until now, any XTable sync involving Delta effectively required a
> > > > > > SparkSession, since the Delta source/target went through
> > > delta-spark's
> > > > > > DeltaLog APIs. The Hudi and Iceberg paths are already pure Java,
> so
> > > > Delta
> > > > > > was the one format anchoring us to Spark. Today the runtime
> options
> > > we
> > > > > ship
> > > > > > (the xtable-utilities CLI and xtable-spark-runtime, #843) both
> > carry
> > > > > Spark.
> > > > > >
> > > > > > With Delta Kernel PR's now merged, Delta can be read and written
> > > > without
> > > > > > Spark. That removes the last hard Spark dependency in the sync
> > path,
> > > > > which
> > > > > > means we can publish a genuinely Spark-free, pure-Java runtime
> jar
> > > > > covering
> > > > > > Hudi, Iceberg, and Delta (via Kernel).
> > > > > >
> > > > > > Why it matters - non-Spark engines
> > > > > >
> > > > > > A Java-only runtime makes XTable embeddable anywhere on the JVM
> > > without
> > > > > > dragging in Spark:
> > > > > >
> > > > > > - Flink jobs syncing metadata in-line
> > > > > > - Trino / Presto plugins and connectors
> > > > > > - Plain Java services, functions, or orchestration steps
> > > > > >
> > > > > > This drastically shrinks the footprint and the
> dependency-conflict
> > > > > surface
> > > > > > versus a Spark bundle, and opens XTable to the large set of users
> > who
> > > > > > aren't on Spark.
> > > > > >
> > > > > > Context
> > > > > >
> > > > > > We discussed this during the community sync this morning and
> agreed
> > > > it's
> > > > > > worth pursuing - the notes capture that with Delta Kernel merged,
> > an
> > > > > > xtable-java-runtime jar can be published with Delta Kernel as
> > > > > > source/target. Full notes:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing
> > > > > >
> > > > > > Questions for the list
> > > > > >
> > > > > > - Do we agree an xtable-java-runtime is worth publishing in the
> > next
> > > > > > release?
> > > > > > - Any concerns about Delta Kernel feature parity for a
> Kernel-only
> > > > > runtime
> > > > > > (e.g. deletion vectors, issue #713)?
> > > > > >
> > > > > > Looking forward to your thoughts.
> > > > > >
> > > > > > Thanks,
> > > > > > Vinish
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] xtable-java-runtime: a Spark-free runtime enabled by Delta Kernel
Posted by Vinish <vi...@gmail.com>.
Hi Tim,
I am withdrawing the xtable-java-runtime proposal. A new module needs
application-specific code, and this one does not have any.
Do you want to revive PR #618, or should someone pick it up against the
current main? It predates Paimon, Parquet and Kernel.
On your point about Delta Kernel, I have filed #886 for the
ITConversionController gap.
https://github.com/apache/incubator-xtable/issues/886.
We will pick it up regardless of how the packaging question lands, since it
is the precondition for making Kernel the default Delta path.
Thanks,
Vinish
On Sun, Jul 19, 2026 08:10 AM, Tim Brown <ti...@gmail.com> wrote:
> If the goal is to make it easier for java applications, I think we should
> just go with the proposal in RFC-2. I have a now outdated implementation
> [1] of this including a bundle validation flow. The proposed jar is
> excluding Paimon for some reason and I think we should aim to support all
> the table formats that are supported. Bundling all of this will just lead
> us back to an overly large jar so it is key to keep things modular.
>
> Generally when people are building java applications, they are building
> bundled jars when they want to run these applications so it doesn't seem
> like a bundled jar is really the requirement here since they will be
> bundling the required dependencies declared by our jar's manifest. The need
> here is to make sure the manifest is as lean as possible so someone that
> just wants delta-kernel and Iceberg will not pull in Hudi, Paimon, or any
> spark dependency.
>
> If there is a secondary concern that the user will already have their own
> version of some of the required dependencies that may conflict with what we
> declare, then it makes sense to also publish our own bundled jars with
> relocated dependencies for these lean modules.
>
> The only reason I can see for adding yet another module to the project is
> if there is some application specific code that needs to be added. This is
> the case for the spark proposal since it will include pre-built solutions
> for how to integrate XTable into existing writers.
>
>
> 1.
>
> https://github.com/apache/incubator-xtable/pull/618/changes#diff-b818b66a60f0848d9b846b5f966ca959be112f27a12694fa9f79002d9ee9e07e
>
> -Tim
>
> On Fri, Jul 17, 2026 at 12:58 PM Vaibhav Kumar <km...@gmail.com>
> wrote:
>
> > Hi Tim,
> >
> >
> >
> > Thanks for the response and sharing the RFC.
> >
> > You're right that we need to be clear on the goal. Let me be precise as
> per
> > our discussion in the last sync call : the goal is not to replace or
> patch
> > the utilities jar. It is to allow XTable to be used as a drop-in library
> in
> > JVM applications that do not run Spark at all — Flink jobs, Trino/Presto
> > plugins, plain Java services. The utility jar cannot serve that goal
> even
> > if we slimmed it down, because it is designed as a CLI tool and bundles
> > Spark by design.
> >
> >
> > This actually lines up directly with the pain points your RFC-2 [1]
> > identifies:
> >
> >
> > ▎ "When running the prebuilt utilities bundle jar, the user will bring
> in
> > all the required
> > ▎ dependencies which currently include three table formats, spark, and
> > hadoop. This results in a
> > ▎ very large jar containing more than the user really needs."
> >
> >
> >
> > ▎ "When building your own jar, you can run into issues with conflicts
> in
> > versions for the
> > ▎ specific table formats you want to use since the user will often
> > already have an implementation
> > ▎ of at least one format on their classpath."
> >
> > An xtable-java-runtime addresses both pain points directly for the
> > non-Spark user segment. It would be a shaded jar with relocated
> > dependencies — the same pattern used in xtable-spark-runtime(#843) —
> > bundling Delta Kernel, Hudi Java client, and Iceberg core with no Spark.
> > Because dependencies are relocated inside the jar, version conflicts on
> the
> > user's classpath are avoided. This is exactly the per-format shaded jar
> > approach your RFC-2 proposes.
> >
> >
> > To your point on jar structure: it would be a shaded fat jar with
> relocated
> > dependencies. Users would consume it via --jars or as a Maven dependency,
> > and the dependency-reduced POM would expose no transitive engine
> > dependencies.
> >
> >
> > On Delta Kernel readiness — you are right that it is not yet integrated
> > into ITConversionController as a first-class path alongside the
> Spark-based
> > Delta tests. The dedicated test suite (ITDeltaKernelConversionSource,
> > TestDeltaKernelReadWriteIntegration, TestDeltaKernelSync) covers the
> > Kernel path in depth, but I agree the shared harness is the right bar.
> > Would it make sense to track the ITConversionController integration as a
> > prerequisite for the java-runtime publish, so we can keep the discussion
> > moving forward while that work gets done in parallel?
> >
> >
> > To summarise — the xtable-java-runtime is not a workaround for the module
> > restructuring RFC. It is one concrete deliverable that RFC-2 leads to,
> > scoped narrowly to a packaging module so it does not depend on the full
> > restructure being complete first.
> >
> >
> > Happy to open a tracking issue for the ITConversionController gap and put
> > together a more detailed design doc if the community agrees with the
> > direction.
> >
> > Thanks,
> >
> > Vaibhav
> >
> >
> >
> > On Tue, Jul 14, 2026 at 5:53 PM Tim Brown <ti...@gmail.com>
> wrote:
> >
> > > I think that we need to figure out what the goal is before proposing
> the
> > > structure. If the goal is to allow a standalone java jar to run
> > > conversions, I would suggest we simply revisit the utilities jar or the
> > web
> > > service for this.
> > >
> > > The main issue with the utilities jar was that it was bundling all
> sorts
> > of
> > > things, including spark. We have a similar issue with the layout of the
> > > project in general where it is hard to pick and choose the lean set of
> > > dependencies that you want for your particular use case. That was the
> > > reason for the RFC for module restructuring [1].
> > >
> > > Before starting any work on this there are a few questions that need to
> > be
> > > answered. What will the xtable-java-runtime jar look like? Will it be a
> > > bundled jar? Will it have relocated dependencies or will it require
> users
> > > to provide the implementation of the table formats?
> > >
> > > Before declaring the Delta Kernel ready for use, I would like to see it
> > > integrated into our integration test suite in ITConversionController
> [2].
> > > It is also worthwhile to look back at the recent issues reported for
> > Delta
> > > conversion and ensure they are also fixed in the Delta Kernel path.
> > >
> > > 1. https://github.com/apache/incubator-xtable/pull/612
> > > 2.
> > >
> > >
> >
> https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java
> > >
> > > On Mon, Jul 13, 2026 at 10:31 PM Vaibhav Kumar <kmickey.1992@gmail.com
> >
> > > wrote:
> > >
> > > > Hey Vinish
> > > >
> > > > +1 for the idea, I can take this up.
> > > >
> > > > On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala <
> > > > vinishreddypannala@gmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to open a discussion on publishing an xtable-java-runtime
> -
> > a
> > > > > lightweight, Spark-free runtime for running XTable metadata sync -
> > now
> > > > that
> > > > > Delta Kernel support has landed on main.
> > > > >
> > > > > Motivation
> > > > >
> > > > > Until now, any XTable sync involving Delta effectively required a
> > > > > SparkSession, since the Delta source/target went through
> > delta-spark's
> > > > > DeltaLog APIs. The Hudi and Iceberg paths are already pure Java, so
> > > Delta
> > > > > was the one format anchoring us to Spark. Today the runtime options
> > we
> > > > ship
> > > > > (the xtable-utilities CLI and xtable-spark-runtime, #843) both
> carry
> > > > Spark.
> > > > >
> > > > > With Delta Kernel PR's now merged, Delta can be read and written
> > > without
> > > > > Spark. That removes the last hard Spark dependency in the sync
> path,
> > > > which
> > > > > means we can publish a genuinely Spark-free, pure-Java runtime jar
> > > > covering
> > > > > Hudi, Iceberg, and Delta (via Kernel).
> > > > >
> > > > > Why it matters - non-Spark engines
> > > > >
> > > > > A Java-only runtime makes XTable embeddable anywhere on the JVM
> > without
> > > > > dragging in Spark:
> > > > >
> > > > > - Flink jobs syncing metadata in-line
> > > > > - Trino / Presto plugins and connectors
> > > > > - Plain Java services, functions, or orchestration steps
> > > > >
> > > > > This drastically shrinks the footprint and the dependency-conflict
> > > > surface
> > > > > versus a Spark bundle, and opens XTable to the large set of users
> who
> > > > > aren't on Spark.
> > > > >
> > > > > Context
> > > > >
> > > > > We discussed this during the community sync this morning and agreed
> > > it's
> > > > > worth pursuing - the notes capture that with Delta Kernel merged,
> an
> > > > > xtable-java-runtime jar can be published with Delta Kernel as
> > > > > source/target. Full notes:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing
> > > > >
> > > > > Questions for the list
> > > > >
> > > > > - Do we agree an xtable-java-runtime is worth publishing in the
> next
> > > > > release?
> > > > > - Any concerns about Delta Kernel feature parity for a Kernel-only
> > > > runtime
> > > > > (e.g. deletion vectors, issue #713)?
> > > > >
> > > > > Looking forward to your thoughts.
> > > > >
> > > > > Thanks,
> > > > > Vinish
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] xtable-java-runtime: a Spark-free runtime enabled by Delta Kernel
Posted by Tim Brown <ti...@gmail.com>.
If the goal is to make it easier for java applications, I think we should
just go with the proposal in RFC-2. I have a now outdated implementation
[1] of this including a bundle validation flow. The proposed jar is
excluding Paimon for some reason and I think we should aim to support all
the table formats that are supported. Bundling all of this will just lead
us back to an overly large jar so it is key to keep things modular.
Generally when people are building java applications, they are building
bundled jars when they want to run these applications so it doesn't seem
like a bundled jar is really the requirement here since they will be
bundling the required dependencies declared by our jar's manifest. The need
here is to make sure the manifest is as lean as possible so someone that
just wants delta-kernel and Iceberg will not pull in Hudi, Paimon, or any
spark dependency.
If there is a secondary concern that the user will already have their own
version of some of the required dependencies that may conflict with what we
declare, then it makes sense to also publish our own bundled jars with
relocated dependencies for these lean modules.
The only reason I can see for adding yet another module to the project is
if there is some application specific code that needs to be added. This is
the case for the spark proposal since it will include pre-built solutions
for how to integrate XTable into existing writers.
1.
https://github.com/apache/incubator-xtable/pull/618/changes#diff-b818b66a60f0848d9b846b5f966ca959be112f27a12694fa9f79002d9ee9e07e
-Tim
On Fri, Jul 17, 2026 at 12:58 PM Vaibhav Kumar <km...@gmail.com>
wrote:
> Hi Tim,
>
>
>
> Thanks for the response and sharing the RFC.
>
> You're right that we need to be clear on the goal. Let me be precise as per
> our discussion in the last sync call : the goal is not to replace or patch
> the utilities jar. It is to allow XTable to be used as a drop-in library in
> JVM applications that do not run Spark at all — Flink jobs, Trino/Presto
> plugins, plain Java services. The utility jar cannot serve that goal even
> if we slimmed it down, because it is designed as a CLI tool and bundles
> Spark by design.
>
>
> This actually lines up directly with the pain points your RFC-2 [1]
> identifies:
>
>
> ▎ "When running the prebuilt utilities bundle jar, the user will bring in
> all the required
> ▎ dependencies which currently include three table formats, spark, and
> hadoop. This results in a
> ▎ very large jar containing more than the user really needs."
>
>
>
> ▎ "When building your own jar, you can run into issues with conflicts in
> versions for the
> ▎ specific table formats you want to use since the user will often
> already have an implementation
> ▎ of at least one format on their classpath."
>
> An xtable-java-runtime addresses both pain points directly for the
> non-Spark user segment. It would be a shaded jar with relocated
> dependencies — the same pattern used in xtable-spark-runtime(#843) —
> bundling Delta Kernel, Hudi Java client, and Iceberg core with no Spark.
> Because dependencies are relocated inside the jar, version conflicts on the
> user's classpath are avoided. This is exactly the per-format shaded jar
> approach your RFC-2 proposes.
>
>
> To your point on jar structure: it would be a shaded fat jar with relocated
> dependencies. Users would consume it via --jars or as a Maven dependency,
> and the dependency-reduced POM would expose no transitive engine
> dependencies.
>
>
> On Delta Kernel readiness — you are right that it is not yet integrated
> into ITConversionController as a first-class path alongside the Spark-based
> Delta tests. The dedicated test suite (ITDeltaKernelConversionSource,
> TestDeltaKernelReadWriteIntegration, TestDeltaKernelSync) covers the
> Kernel path in depth, but I agree the shared harness is the right bar.
> Would it make sense to track the ITConversionController integration as a
> prerequisite for the java-runtime publish, so we can keep the discussion
> moving forward while that work gets done in parallel?
>
>
> To summarise — the xtable-java-runtime is not a workaround for the module
> restructuring RFC. It is one concrete deliverable that RFC-2 leads to,
> scoped narrowly to a packaging module so it does not depend on the full
> restructure being complete first.
>
>
> Happy to open a tracking issue for the ITConversionController gap and put
> together a more detailed design doc if the community agrees with the
> direction.
>
> Thanks,
>
> Vaibhav
>
>
>
> On Tue, Jul 14, 2026 at 5:53 PM Tim Brown <ti...@gmail.com> wrote:
>
> > I think that we need to figure out what the goal is before proposing the
> > structure. If the goal is to allow a standalone java jar to run
> > conversions, I would suggest we simply revisit the utilities jar or the
> web
> > service for this.
> >
> > The main issue with the utilities jar was that it was bundling all sorts
> of
> > things, including spark. We have a similar issue with the layout of the
> > project in general where it is hard to pick and choose the lean set of
> > dependencies that you want for your particular use case. That was the
> > reason for the RFC for module restructuring [1].
> >
> > Before starting any work on this there are a few questions that need to
> be
> > answered. What will the xtable-java-runtime jar look like? Will it be a
> > bundled jar? Will it have relocated dependencies or will it require users
> > to provide the implementation of the table formats?
> >
> > Before declaring the Delta Kernel ready for use, I would like to see it
> > integrated into our integration test suite in ITConversionController [2].
> > It is also worthwhile to look back at the recent issues reported for
> Delta
> > conversion and ensure they are also fixed in the Delta Kernel path.
> >
> > 1. https://github.com/apache/incubator-xtable/pull/612
> > 2.
> >
> >
> https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java
> >
> > On Mon, Jul 13, 2026 at 10:31 PM Vaibhav Kumar <km...@gmail.com>
> > wrote:
> >
> > > Hey Vinish
> > >
> > > +1 for the idea, I can take this up.
> > >
> > > On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala <
> > > vinishreddypannala@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to open a discussion on publishing an xtable-java-runtime -
> a
> > > > lightweight, Spark-free runtime for running XTable metadata sync -
> now
> > > that
> > > > Delta Kernel support has landed on main.
> > > >
> > > > Motivation
> > > >
> > > > Until now, any XTable sync involving Delta effectively required a
> > > > SparkSession, since the Delta source/target went through
> delta-spark's
> > > > DeltaLog APIs. The Hudi and Iceberg paths are already pure Java, so
> > Delta
> > > > was the one format anchoring us to Spark. Today the runtime options
> we
> > > ship
> > > > (the xtable-utilities CLI and xtable-spark-runtime, #843) both carry
> > > Spark.
> > > >
> > > > With Delta Kernel PR's now merged, Delta can be read and written
> > without
> > > > Spark. That removes the last hard Spark dependency in the sync path,
> > > which
> > > > means we can publish a genuinely Spark-free, pure-Java runtime jar
> > > covering
> > > > Hudi, Iceberg, and Delta (via Kernel).
> > > >
> > > > Why it matters - non-Spark engines
> > > >
> > > > A Java-only runtime makes XTable embeddable anywhere on the JVM
> without
> > > > dragging in Spark:
> > > >
> > > > - Flink jobs syncing metadata in-line
> > > > - Trino / Presto plugins and connectors
> > > > - Plain Java services, functions, or orchestration steps
> > > >
> > > > This drastically shrinks the footprint and the dependency-conflict
> > > surface
> > > > versus a Spark bundle, and opens XTable to the large set of users who
> > > > aren't on Spark.
> > > >
> > > > Context
> > > >
> > > > We discussed this during the community sync this morning and agreed
> > it's
> > > > worth pursuing - the notes capture that with Delta Kernel merged, an
> > > > xtable-java-runtime jar can be published with Delta Kernel as
> > > > source/target. Full notes:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing
> > > >
> > > > Questions for the list
> > > >
> > > > - Do we agree an xtable-java-runtime is worth publishing in the next
> > > > release?
> > > > - Any concerns about Delta Kernel feature parity for a Kernel-only
> > > runtime
> > > > (e.g. deletion vectors, issue #713)?
> > > >
> > > > Looking forward to your thoughts.
> > > >
> > > > Thanks,
> > > > Vinish
> > > >
> > >
> >
>
Re: [DISCUSS] xtable-java-runtime: a Spark-free runtime enabled by Delta Kernel
Posted by Vaibhav Kumar <km...@gmail.com>.
Hi Tim,
Thanks for the response and sharing the RFC.
You're right that we need to be clear on the goal. Let me be precise as per
our discussion in the last sync call : the goal is not to replace or patch
the utilities jar. It is to allow XTable to be used as a drop-in library in
JVM applications that do not run Spark at all — Flink jobs, Trino/Presto
plugins, plain Java services. The utility jar cannot serve that goal even
if we slimmed it down, because it is designed as a CLI tool and bundles
Spark by design.
This actually lines up directly with the pain points your RFC-2 [1]
identifies:
▎ "When running the prebuilt utilities bundle jar, the user will bring in
all the required
▎ dependencies which currently include three table formats, spark, and
hadoop. This results in a
▎ very large jar containing more than the user really needs."
▎ "When building your own jar, you can run into issues with conflicts in
versions for the
▎ specific table formats you want to use since the user will often
already have an implementation
▎ of at least one format on their classpath."
An xtable-java-runtime addresses both pain points directly for the
non-Spark user segment. It would be a shaded jar with relocated
dependencies — the same pattern used in xtable-spark-runtime(#843) —
bundling Delta Kernel, Hudi Java client, and Iceberg core with no Spark.
Because dependencies are relocated inside the jar, version conflicts on the
user's classpath are avoided. This is exactly the per-format shaded jar
approach your RFC-2 proposes.
To your point on jar structure: it would be a shaded fat jar with relocated
dependencies. Users would consume it via --jars or as a Maven dependency,
and the dependency-reduced POM would expose no transitive engine
dependencies.
On Delta Kernel readiness — you are right that it is not yet integrated
into ITConversionController as a first-class path alongside the Spark-based
Delta tests. The dedicated test suite (ITDeltaKernelConversionSource,
TestDeltaKernelReadWriteIntegration, TestDeltaKernelSync) covers the
Kernel path in depth, but I agree the shared harness is the right bar.
Would it make sense to track the ITConversionController integration as a
prerequisite for the java-runtime publish, so we can keep the discussion
moving forward while that work gets done in parallel?
To summarise — the xtable-java-runtime is not a workaround for the module
restructuring RFC. It is one concrete deliverable that RFC-2 leads to,
scoped narrowly to a packaging module so it does not depend on the full
restructure being complete first.
Happy to open a tracking issue for the ITConversionController gap and put
together a more detailed design doc if the community agrees with the
direction.
Thanks,
Vaibhav
On Tue, Jul 14, 2026 at 5:53 PM Tim Brown <ti...@gmail.com> wrote:
> I think that we need to figure out what the goal is before proposing the
> structure. If the goal is to allow a standalone java jar to run
> conversions, I would suggest we simply revisit the utilities jar or the web
> service for this.
>
> The main issue with the utilities jar was that it was bundling all sorts of
> things, including spark. We have a similar issue with the layout of the
> project in general where it is hard to pick and choose the lean set of
> dependencies that you want for your particular use case. That was the
> reason for the RFC for module restructuring [1].
>
> Before starting any work on this there are a few questions that need to be
> answered. What will the xtable-java-runtime jar look like? Will it be a
> bundled jar? Will it have relocated dependencies or will it require users
> to provide the implementation of the table formats?
>
> Before declaring the Delta Kernel ready for use, I would like to see it
> integrated into our integration test suite in ITConversionController [2].
> It is also worthwhile to look back at the recent issues reported for Delta
> conversion and ensure they are also fixed in the Delta Kernel path.
>
> 1. https://github.com/apache/incubator-xtable/pull/612
> 2.
>
> https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java
>
> On Mon, Jul 13, 2026 at 10:31 PM Vaibhav Kumar <km...@gmail.com>
> wrote:
>
> > Hey Vinish
> >
> > +1 for the idea, I can take this up.
> >
> > On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala <
> > vinishreddypannala@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to open a discussion on publishing an xtable-java-runtime - a
> > > lightweight, Spark-free runtime for running XTable metadata sync - now
> > that
> > > Delta Kernel support has landed on main.
> > >
> > > Motivation
> > >
> > > Until now, any XTable sync involving Delta effectively required a
> > > SparkSession, since the Delta source/target went through delta-spark's
> > > DeltaLog APIs. The Hudi and Iceberg paths are already pure Java, so
> Delta
> > > was the one format anchoring us to Spark. Today the runtime options we
> > ship
> > > (the xtable-utilities CLI and xtable-spark-runtime, #843) both carry
> > Spark.
> > >
> > > With Delta Kernel PR's now merged, Delta can be read and written
> without
> > > Spark. That removes the last hard Spark dependency in the sync path,
> > which
> > > means we can publish a genuinely Spark-free, pure-Java runtime jar
> > covering
> > > Hudi, Iceberg, and Delta (via Kernel).
> > >
> > > Why it matters - non-Spark engines
> > >
> > > A Java-only runtime makes XTable embeddable anywhere on the JVM without
> > > dragging in Spark:
> > >
> > > - Flink jobs syncing metadata in-line
> > > - Trino / Presto plugins and connectors
> > > - Plain Java services, functions, or orchestration steps
> > >
> > > This drastically shrinks the footprint and the dependency-conflict
> > surface
> > > versus a Spark bundle, and opens XTable to the large set of users who
> > > aren't on Spark.
> > >
> > > Context
> > >
> > > We discussed this during the community sync this morning and agreed
> it's
> > > worth pursuing - the notes capture that with Delta Kernel merged, an
> > > xtable-java-runtime jar can be published with Delta Kernel as
> > > source/target. Full notes:
> > >
> > >
> >
> https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing
> > >
> > > Questions for the list
> > >
> > > - Do we agree an xtable-java-runtime is worth publishing in the next
> > > release?
> > > - Any concerns about Delta Kernel feature parity for a Kernel-only
> > runtime
> > > (e.g. deletion vectors, issue #713)?
> > >
> > > Looking forward to your thoughts.
> > >
> > > Thanks,
> > > Vinish
> > >
> >
>
Re: [DISCUSS] xtable-java-runtime: a Spark-free runtime enabled by Delta Kernel
Posted by Tim Brown <ti...@gmail.com>.
I think that we need to figure out what the goal is before proposing the
structure. If the goal is to allow a standalone java jar to run
conversions, I would suggest we simply revisit the utilities jar or the web
service for this.
The main issue with the utilities jar was that it was bundling all sorts of
things, including spark. We have a similar issue with the layout of the
project in general where it is hard to pick and choose the lean set of
dependencies that you want for your particular use case. That was the
reason for the RFC for module restructuring [1].
Before starting any work on this there are a few questions that need to be
answered. What will the xtable-java-runtime jar look like? Will it be a
bundled jar? Will it have relocated dependencies or will it require users
to provide the implementation of the table formats?
Before declaring the Delta Kernel ready for use, I would like to see it
integrated into our integration test suite in ITConversionController [2].
It is also worthwhile to look back at the recent issues reported for Delta
conversion and ensure they are also fixed in the Delta Kernel path.
1. https://github.com/apache/incubator-xtable/pull/612
2.
https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java
On Mon, Jul 13, 2026 at 10:31 PM Vaibhav Kumar <km...@gmail.com>
wrote:
> Hey Vinish
>
> +1 for the idea, I can take this up.
>
> On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala <
> vinishreddypannala@gmail.com> wrote:
>
> > Hi all,
> >
> > I'd like to open a discussion on publishing an xtable-java-runtime - a
> > lightweight, Spark-free runtime for running XTable metadata sync - now
> that
> > Delta Kernel support has landed on main.
> >
> > Motivation
> >
> > Until now, any XTable sync involving Delta effectively required a
> > SparkSession, since the Delta source/target went through delta-spark's
> > DeltaLog APIs. The Hudi and Iceberg paths are already pure Java, so Delta
> > was the one format anchoring us to Spark. Today the runtime options we
> ship
> > (the xtable-utilities CLI and xtable-spark-runtime, #843) both carry
> Spark.
> >
> > With Delta Kernel PR's now merged, Delta can be read and written without
> > Spark. That removes the last hard Spark dependency in the sync path,
> which
> > means we can publish a genuinely Spark-free, pure-Java runtime jar
> covering
> > Hudi, Iceberg, and Delta (via Kernel).
> >
> > Why it matters - non-Spark engines
> >
> > A Java-only runtime makes XTable embeddable anywhere on the JVM without
> > dragging in Spark:
> >
> > - Flink jobs syncing metadata in-line
> > - Trino / Presto plugins and connectors
> > - Plain Java services, functions, or orchestration steps
> >
> > This drastically shrinks the footprint and the dependency-conflict
> surface
> > versus a Spark bundle, and opens XTable to the large set of users who
> > aren't on Spark.
> >
> > Context
> >
> > We discussed this during the community sync this morning and agreed it's
> > worth pursuing - the notes capture that with Delta Kernel merged, an
> > xtable-java-runtime jar can be published with Delta Kernel as
> > source/target. Full notes:
> >
> >
> https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing
> >
> > Questions for the list
> >
> > - Do we agree an xtable-java-runtime is worth publishing in the next
> > release?
> > - Any concerns about Delta Kernel feature parity for a Kernel-only
> runtime
> > (e.g. deletion vectors, issue #713)?
> >
> > Looking forward to your thoughts.
> >
> > Thanks,
> > Vinish
> >
>
Re: [DISCUSS] xtable-java-runtime: a Spark-free runtime enabled by Delta Kernel
Posted by Vaibhav Kumar <km...@gmail.com>.
Hey Vinish
+1 for the idea, I can take this up.
On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala <
vinishreddypannala@gmail.com> wrote:
> Hi all,
>
> I'd like to open a discussion on publishing an xtable-java-runtime - a
> lightweight, Spark-free runtime for running XTable metadata sync - now that
> Delta Kernel support has landed on main.
>
> Motivation
>
> Until now, any XTable sync involving Delta effectively required a
> SparkSession, since the Delta source/target went through delta-spark's
> DeltaLog APIs. The Hudi and Iceberg paths are already pure Java, so Delta
> was the one format anchoring us to Spark. Today the runtime options we ship
> (the xtable-utilities CLI and xtable-spark-runtime, #843) both carry Spark.
>
> With Delta Kernel PR's now merged, Delta can be read and written without
> Spark. That removes the last hard Spark dependency in the sync path, which
> means we can publish a genuinely Spark-free, pure-Java runtime jar covering
> Hudi, Iceberg, and Delta (via Kernel).
>
> Why it matters - non-Spark engines
>
> A Java-only runtime makes XTable embeddable anywhere on the JVM without
> dragging in Spark:
>
> - Flink jobs syncing metadata in-line
> - Trino / Presto plugins and connectors
> - Plain Java services, functions, or orchestration steps
>
> This drastically shrinks the footprint and the dependency-conflict surface
> versus a Spark bundle, and opens XTable to the large set of users who
> aren't on Spark.
>
> Context
>
> We discussed this during the community sync this morning and agreed it's
> worth pursuing - the notes capture that with Delta Kernel merged, an
> xtable-java-runtime jar can be published with Delta Kernel as
> source/target. Full notes:
>
> https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing
>
> Questions for the list
>
> - Do we agree an xtable-java-runtime is worth publishing in the next
> release?
> - Any concerns about Delta Kernel feature parity for a Kernel-only runtime
> (e.g. deletion vectors, issue #713)?
>
> Looking forward to your thoughts.
>
> Thanks,
> Vinish
>