You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Cancai Cai <ca...@gmail.com> on 2026/06/04 06:19:10 UTC
Subject: [DISCUSS] Removing Gandiva from the Arrow adapter
Hi all,
I would like to discuss the future of Gandiva in Calcite's Arrow adapter.
My preferred long-term direction is to remove the Gandiva dependency from
the adapter.
The current adapter uses Arrow Java to read Arrow data, but relies on
Gandiva `Projector` and `Filter` for projection and filter execution.
Gandiva is a native LLVM-based runtime, and the Java module is a wrapper
around that native implementation. As a result, basic Arrow adapter queries
depend on native libraries, LLVM compatibility, platform packaging, and JDK
baseline details.
This has become a practical maintenance problem when thinking about Arrow
dependency upgrades.
The upgrade problem is not limited to Java bytecode compatibility. In our
experiments, newer Arrow versions failed at different layers. Arrow 18
requires a newer Java baseline than Calcite currently supports in its JDK 8
jobs. Arrow 17 and 16.1 still use Java 8 class files, but can hit Java
runtime API incompatibilities on JDK 8, such as `ByteBuffer.flip():
ByteBuffer`. Arrow 16.0 avoids that Java runtime issue, but exposed Gandiva
native / LLVM symbol issues on Linux CI.
This means that as long as `arrow-gandiva` is required for the adapter's
correctness path, upgrading the Arrow Java vector layer also requires
validating the native Gandiva stack across all CI platforms. Even when
`arrow-vector` itself is usable, `arrow-gandiva` can still block the
upgrade.
For that reason, I think the adapter should make projection/filter
correctness independent of Gandiva first. Once the Java correctness path is
in place, Arrow vector upgrades can be evaluated separately from Gandiva
native compatibility.
The direction I have in mind is a pure Java correctness path for the Arrow
adapter:
* read Arrow data with `ArrowFileReader`, `VectorSchemaRoot`, and
`ValueVector`;
* execute simple projections by reading selected vectors directly;
* execute the simple filters currently translated by `ArrowTranslator` with
a Java evaluator;
* leave expressions that are not pushed into the adapter to Calcite's
normal Enumerable / code generation path.
With that model, Gandiva would no longer be required for correctness. A
staged migration could be:
1. Move no-filter simple projection away from Gandiva.
2. Add Java evaluation for the simple filter subset currently supported by
`ArrowTranslator`.
3. Validate that existing Arrow adapter tests pass without invoking Gandiva.
4. Remove `arrow-gandiva` from the adapter dependency set once the Java
path covers the current behavior.
The tradeoff is that Gandiva may be faster for supported expressions. But
for this adapter, I think correctness, portability, and dependency
stability should come first. If acceleration is needed later, it can be
discussed separately.
Does this direction make sense to the community? Are there current use
cases that depend on Gandiva pushdown strongly enough that we should keep
the native dependency?
Thanks,
Cancai
Re: Subject: [DISCUSS] Removing Gandiva from the Arrow adapter
Posted by Cancai Cai <ca...@gmail.com>.
Restore https://github.com/apache/calcite/pull/5025.
This is the first time I've heard of this issue with Git, sorry.
Best wishes,
Cancai
Cancai Cai <ca...@gmail.com> 于2026年6月9日周二 11:48写道:
> I'm very sorry about my mistake; it was because I didn't look carefully
> enough when writing the code in the AI. I'm very sorry for that.
>
> Since I'm on vacation this week, I will resolve this issue next week.
> Best wishes.
> Cancai
>
> Sergey Nuyanzin <sn...@gmail.com> 于2026年6月8日周一 16:01写道:
>
>> Hi everyone
>>
>> I noticed that removal of Gandiva PR/commit[1] made extra more steps
>> than probably it should
>>
>> for instance it implicitly completely reverted fixe for
>> CALCITE-7562[2], CALCITE-7567[3], CALCITE-7578[4], CALCITE-7588[5]
>>
>> Was it intentional?
>>
>>
>> [1] https://github.com/apache/calcite/pull/4992
>> [2] https://issues.apache.org/jira/browse/CALCITE-7562
>> [3] https://issues.apache.org/jira/browse/CALCITE-7567
>> [4] https://issues.apache.org/jira/browse/CALCITE-7578
>> [5] https://issues.apache.org/jira/browse/CALCITE-7588
>>
>> On Thu, Jun 4, 2026 at 4:53 PM Julian Hyde <jh...@gmail.com>
>> wrote:
>> >
>> > +1
>> >
>> > > On Jun 4, 2026, at 12:57 AM, Alessandro Solimando <
>> alessandro.solimando@gmail.com> wrote:
>> > >
>> > > If it's not supported anymore it's already a good enough argument to
>> > > replace it.
>> > >
>> > > The proposed plan looks reasonable to me.
>> > >
>> > > Alessandro
>> > >
>> > >> On Thu, Jun 4, 2026, 09:45 jensen <cz...@163.com> wrote:
>> > >>
>> > >> I vaguely recall that Arrow's official website also discourages the
>> use of
>> > >> Gandiva, and Gandiva is no longer maintained. If possible, I think we
>> > >> should remove this dependency.
>> > >>
>> > >>
>> > >>
>> > >> Best regards,
>> > >>
>> > >> Zhen
>> > >>
>> > >> ---- Replied Message ----
>> > >> | From | Cancai Cai<ca...@gmail.com> |
>> > >> | Date | 6/4/2026 14:20 |
>> > >> | To | <de...@calcite.apache.org> |
>> > >> | Subject | Subject: [DISCUSS] Removing Gandiva from the Arrow
>> adapter |
>> > >> Hi all,
>> > >>
>> > >> I would like to discuss the future of Gandiva in Calcite's Arrow
>> adapter.
>> > >> My preferred long-term direction is to remove the Gandiva dependency
>> from
>> > >> the adapter.
>> > >>
>> > >> The current adapter uses Arrow Java to read Arrow data, but relies on
>> > >> Gandiva `Projector` and `Filter` for projection and filter execution.
>> > >> Gandiva is a native LLVM-based runtime, and the Java module is a
>> wrapper
>> > >> around that native implementation. As a result, basic Arrow adapter
>> queries
>> > >> depend on native libraries, LLVM compatibility, platform packaging,
>> and JDK
>> > >> baseline details.
>> > >>
>> > >> This has become a practical maintenance problem when thinking about
>> Arrow
>> > >> dependency upgrades.
>> > >>
>> > >> The upgrade problem is not limited to Java bytecode compatibility.
>> In our
>> > >> experiments, newer Arrow versions failed at different layers. Arrow
>> 18
>> > >> requires a newer Java baseline than Calcite currently supports in
>> its JDK 8
>> > >> jobs. Arrow 17 and 16.1 still use Java 8 class files, but can hit
>> Java
>> > >> runtime API incompatibilities on JDK 8, such as `ByteBuffer.flip():
>> > >> ByteBuffer`. Arrow 16.0 avoids that Java runtime issue, but exposed
>> Gandiva
>> > >> native / LLVM symbol issues on Linux CI.
>> > >>
>> > >> This means that as long as `arrow-gandiva` is required for the
>> adapter's
>> > >> correctness path, upgrading the Arrow Java vector layer also requires
>> > >> validating the native Gandiva stack across all CI platforms. Even
>> when
>> > >> `arrow-vector` itself is usable, `arrow-gandiva` can still block the
>> > >> upgrade.
>> > >>
>> > >> For that reason, I think the adapter should make projection/filter
>> > >> correctness independent of Gandiva first. Once the Java correctness
>> path is
>> > >> in place, Arrow vector upgrades can be evaluated separately from
>> Gandiva
>> > >> native compatibility.
>> > >>
>> > >> The direction I have in mind is a pure Java correctness path for the
>> Arrow
>> > >> adapter:
>> > >>
>> > >> * read Arrow data with `ArrowFileReader`, `VectorSchemaRoot`, and
>> > >> `ValueVector`;
>> > >> * execute simple projections by reading selected vectors directly;
>> > >> * execute the simple filters currently translated by
>> `ArrowTranslator` with
>> > >> a Java evaluator;
>> > >> * leave expressions that are not pushed into the adapter to Calcite's
>> > >> normal Enumerable / code generation path.
>> > >>
>> > >> With that model, Gandiva would no longer be required for
>> correctness. A
>> > >> staged migration could be:
>> > >>
>> > >> 1. Move no-filter simple projection away from Gandiva.
>> > >> 2. Add Java evaluation for the simple filter subset currently
>> supported by
>> > >> `ArrowTranslator`.
>> > >> 3. Validate that existing Arrow adapter tests pass without invoking
>> > >> Gandiva.
>> > >> 4. Remove `arrow-gandiva` from the adapter dependency set once the
>> Java
>> > >> path covers the current behavior.
>> > >>
>> > >> The tradeoff is that Gandiva may be faster for supported
>> expressions. But
>> > >> for this adapter, I think correctness, portability, and dependency
>> > >> stability should come first. If acceleration is needed later, it can
>> be
>> > >> discussed separately.
>> > >>
>> > >> Does this direction make sense to the community? Are there current
>> use
>> > >> cases that depend on Gandiva pushdown strongly enough that we should
>> keep
>> > >> the native dependency?
>> > >>
>> > >> Thanks,
>> > >> Cancai
>> > >>
>>
>>
>>
>> --
>> Best regards,
>> Sergey
>>
>
Re: Subject: [DISCUSS] Removing Gandiva from the Arrow adapter
Posted by Cancai Cai <ca...@gmail.com>.
I'm very sorry about my mistake; it was because I didn't look carefully
enough when writing the code in the AI. I'm very sorry for that.
Since I'm on vacation this week, I will resolve this issue next week.
Best wishes.
Cancai
Sergey Nuyanzin <sn...@gmail.com> 于2026年6月8日周一 16:01写道:
> Hi everyone
>
> I noticed that removal of Gandiva PR/commit[1] made extra more steps
> than probably it should
>
> for instance it implicitly completely reverted fixe for
> CALCITE-7562[2], CALCITE-7567[3], CALCITE-7578[4], CALCITE-7588[5]
>
> Was it intentional?
>
>
> [1] https://github.com/apache/calcite/pull/4992
> [2] https://issues.apache.org/jira/browse/CALCITE-7562
> [3] https://issues.apache.org/jira/browse/CALCITE-7567
> [4] https://issues.apache.org/jira/browse/CALCITE-7578
> [5] https://issues.apache.org/jira/browse/CALCITE-7588
>
> On Thu, Jun 4, 2026 at 4:53 PM Julian Hyde <jh...@gmail.com> wrote:
> >
> > +1
> >
> > > On Jun 4, 2026, at 12:57 AM, Alessandro Solimando <
> alessandro.solimando@gmail.com> wrote:
> > >
> > > If it's not supported anymore it's already a good enough argument to
> > > replace it.
> > >
> > > The proposed plan looks reasonable to me.
> > >
> > > Alessandro
> > >
> > >> On Thu, Jun 4, 2026, 09:45 jensen <cz...@163.com> wrote:
> > >>
> > >> I vaguely recall that Arrow's official website also discourages the
> use of
> > >> Gandiva, and Gandiva is no longer maintained. If possible, I think we
> > >> should remove this dependency.
> > >>
> > >>
> > >>
> > >> Best regards,
> > >>
> > >> Zhen
> > >>
> > >> ---- Replied Message ----
> > >> | From | Cancai Cai<ca...@gmail.com> |
> > >> | Date | 6/4/2026 14:20 |
> > >> | To | <de...@calcite.apache.org> |
> > >> | Subject | Subject: [DISCUSS] Removing Gandiva from the Arrow
> adapter |
> > >> Hi all,
> > >>
> > >> I would like to discuss the future of Gandiva in Calcite's Arrow
> adapter.
> > >> My preferred long-term direction is to remove the Gandiva dependency
> from
> > >> the adapter.
> > >>
> > >> The current adapter uses Arrow Java to read Arrow data, but relies on
> > >> Gandiva `Projector` and `Filter` for projection and filter execution.
> > >> Gandiva is a native LLVM-based runtime, and the Java module is a
> wrapper
> > >> around that native implementation. As a result, basic Arrow adapter
> queries
> > >> depend on native libraries, LLVM compatibility, platform packaging,
> and JDK
> > >> baseline details.
> > >>
> > >> This has become a practical maintenance problem when thinking about
> Arrow
> > >> dependency upgrades.
> > >>
> > >> The upgrade problem is not limited to Java bytecode compatibility. In
> our
> > >> experiments, newer Arrow versions failed at different layers. Arrow 18
> > >> requires a newer Java baseline than Calcite currently supports in its
> JDK 8
> > >> jobs. Arrow 17 and 16.1 still use Java 8 class files, but can hit Java
> > >> runtime API incompatibilities on JDK 8, such as `ByteBuffer.flip():
> > >> ByteBuffer`. Arrow 16.0 avoids that Java runtime issue, but exposed
> Gandiva
> > >> native / LLVM symbol issues on Linux CI.
> > >>
> > >> This means that as long as `arrow-gandiva` is required for the
> adapter's
> > >> correctness path, upgrading the Arrow Java vector layer also requires
> > >> validating the native Gandiva stack across all CI platforms. Even when
> > >> `arrow-vector` itself is usable, `arrow-gandiva` can still block the
> > >> upgrade.
> > >>
> > >> For that reason, I think the adapter should make projection/filter
> > >> correctness independent of Gandiva first. Once the Java correctness
> path is
> > >> in place, Arrow vector upgrades can be evaluated separately from
> Gandiva
> > >> native compatibility.
> > >>
> > >> The direction I have in mind is a pure Java correctness path for the
> Arrow
> > >> adapter:
> > >>
> > >> * read Arrow data with `ArrowFileReader`, `VectorSchemaRoot`, and
> > >> `ValueVector`;
> > >> * execute simple projections by reading selected vectors directly;
> > >> * execute the simple filters currently translated by
> `ArrowTranslator` with
> > >> a Java evaluator;
> > >> * leave expressions that are not pushed into the adapter to Calcite's
> > >> normal Enumerable / code generation path.
> > >>
> > >> With that model, Gandiva would no longer be required for correctness.
> A
> > >> staged migration could be:
> > >>
> > >> 1. Move no-filter simple projection away from Gandiva.
> > >> 2. Add Java evaluation for the simple filter subset currently
> supported by
> > >> `ArrowTranslator`.
> > >> 3. Validate that existing Arrow adapter tests pass without invoking
> > >> Gandiva.
> > >> 4. Remove `arrow-gandiva` from the adapter dependency set once the
> Java
> > >> path covers the current behavior.
> > >>
> > >> The tradeoff is that Gandiva may be faster for supported expressions.
> But
> > >> for this adapter, I think correctness, portability, and dependency
> > >> stability should come first. If acceleration is needed later, it can
> be
> > >> discussed separately.
> > >>
> > >> Does this direction make sense to the community? Are there current use
> > >> cases that depend on Gandiva pushdown strongly enough that we should
> keep
> > >> the native dependency?
> > >>
> > >> Thanks,
> > >> Cancai
> > >>
>
>
>
> --
> Best regards,
> Sergey
>
Re: Subject: [DISCUSS] Removing Gandiva from the Arrow adapter
Posted by Sergey Nuyanzin <sn...@gmail.com>.
Hi everyone
I noticed that removal of Gandiva PR/commit[1] made extra more steps
than probably it should
for instance it implicitly completely reverted fixe for
CALCITE-7562[2], CALCITE-7567[3], CALCITE-7578[4], CALCITE-7588[5]
Was it intentional?
[1] https://github.com/apache/calcite/pull/4992
[2] https://issues.apache.org/jira/browse/CALCITE-7562
[3] https://issues.apache.org/jira/browse/CALCITE-7567
[4] https://issues.apache.org/jira/browse/CALCITE-7578
[5] https://issues.apache.org/jira/browse/CALCITE-7588
On Thu, Jun 4, 2026 at 4:53 PM Julian Hyde <jh...@gmail.com> wrote:
>
> +1
>
> > On Jun 4, 2026, at 12:57 AM, Alessandro Solimando <al...@gmail.com> wrote:
> >
> > If it's not supported anymore it's already a good enough argument to
> > replace it.
> >
> > The proposed plan looks reasonable to me.
> >
> > Alessandro
> >
> >> On Thu, Jun 4, 2026, 09:45 jensen <cz...@163.com> wrote:
> >>
> >> I vaguely recall that Arrow's official website also discourages the use of
> >> Gandiva, and Gandiva is no longer maintained. If possible, I think we
> >> should remove this dependency.
> >>
> >>
> >>
> >> Best regards,
> >>
> >> Zhen
> >>
> >> ---- Replied Message ----
> >> | From | Cancai Cai<ca...@gmail.com> |
> >> | Date | 6/4/2026 14:20 |
> >> | To | <de...@calcite.apache.org> |
> >> | Subject | Subject: [DISCUSS] Removing Gandiva from the Arrow adapter |
> >> Hi all,
> >>
> >> I would like to discuss the future of Gandiva in Calcite's Arrow adapter.
> >> My preferred long-term direction is to remove the Gandiva dependency from
> >> the adapter.
> >>
> >> The current adapter uses Arrow Java to read Arrow data, but relies on
> >> Gandiva `Projector` and `Filter` for projection and filter execution.
> >> Gandiva is a native LLVM-based runtime, and the Java module is a wrapper
> >> around that native implementation. As a result, basic Arrow adapter queries
> >> depend on native libraries, LLVM compatibility, platform packaging, and JDK
> >> baseline details.
> >>
> >> This has become a practical maintenance problem when thinking about Arrow
> >> dependency upgrades.
> >>
> >> The upgrade problem is not limited to Java bytecode compatibility. In our
> >> experiments, newer Arrow versions failed at different layers. Arrow 18
> >> requires a newer Java baseline than Calcite currently supports in its JDK 8
> >> jobs. Arrow 17 and 16.1 still use Java 8 class files, but can hit Java
> >> runtime API incompatibilities on JDK 8, such as `ByteBuffer.flip():
> >> ByteBuffer`. Arrow 16.0 avoids that Java runtime issue, but exposed Gandiva
> >> native / LLVM symbol issues on Linux CI.
> >>
> >> This means that as long as `arrow-gandiva` is required for the adapter's
> >> correctness path, upgrading the Arrow Java vector layer also requires
> >> validating the native Gandiva stack across all CI platforms. Even when
> >> `arrow-vector` itself is usable, `arrow-gandiva` can still block the
> >> upgrade.
> >>
> >> For that reason, I think the adapter should make projection/filter
> >> correctness independent of Gandiva first. Once the Java correctness path is
> >> in place, Arrow vector upgrades can be evaluated separately from Gandiva
> >> native compatibility.
> >>
> >> The direction I have in mind is a pure Java correctness path for the Arrow
> >> adapter:
> >>
> >> * read Arrow data with `ArrowFileReader`, `VectorSchemaRoot`, and
> >> `ValueVector`;
> >> * execute simple projections by reading selected vectors directly;
> >> * execute the simple filters currently translated by `ArrowTranslator` with
> >> a Java evaluator;
> >> * leave expressions that are not pushed into the adapter to Calcite's
> >> normal Enumerable / code generation path.
> >>
> >> With that model, Gandiva would no longer be required for correctness. A
> >> staged migration could be:
> >>
> >> 1. Move no-filter simple projection away from Gandiva.
> >> 2. Add Java evaluation for the simple filter subset currently supported by
> >> `ArrowTranslator`.
> >> 3. Validate that existing Arrow adapter tests pass without invoking
> >> Gandiva.
> >> 4. Remove `arrow-gandiva` from the adapter dependency set once the Java
> >> path covers the current behavior.
> >>
> >> The tradeoff is that Gandiva may be faster for supported expressions. But
> >> for this adapter, I think correctness, portability, and dependency
> >> stability should come first. If acceleration is needed later, it can be
> >> discussed separately.
> >>
> >> Does this direction make sense to the community? Are there current use
> >> cases that depend on Gandiva pushdown strongly enough that we should keep
> >> the native dependency?
> >>
> >> Thanks,
> >> Cancai
> >>
--
Best regards,
Sergey
Re: Subject: [DISCUSS] Removing Gandiva from the Arrow adapter
Posted by Julian Hyde <jh...@gmail.com>.
+1
> On Jun 4, 2026, at 12:57 AM, Alessandro Solimando <al...@gmail.com> wrote:
>
> If it's not supported anymore it's already a good enough argument to
> replace it.
>
> The proposed plan looks reasonable to me.
>
> Alessandro
>
>> On Thu, Jun 4, 2026, 09:45 jensen <cz...@163.com> wrote:
>>
>> I vaguely recall that Arrow's official website also discourages the use of
>> Gandiva, and Gandiva is no longer maintained. If possible, I think we
>> should remove this dependency.
>>
>>
>>
>> Best regards,
>>
>> Zhen
>>
>> ---- Replied Message ----
>> | From | Cancai Cai<ca...@gmail.com> |
>> | Date | 6/4/2026 14:20 |
>> | To | <de...@calcite.apache.org> |
>> | Subject | Subject: [DISCUSS] Removing Gandiva from the Arrow adapter |
>> Hi all,
>>
>> I would like to discuss the future of Gandiva in Calcite's Arrow adapter.
>> My preferred long-term direction is to remove the Gandiva dependency from
>> the adapter.
>>
>> The current adapter uses Arrow Java to read Arrow data, but relies on
>> Gandiva `Projector` and `Filter` for projection and filter execution.
>> Gandiva is a native LLVM-based runtime, and the Java module is a wrapper
>> around that native implementation. As a result, basic Arrow adapter queries
>> depend on native libraries, LLVM compatibility, platform packaging, and JDK
>> baseline details.
>>
>> This has become a practical maintenance problem when thinking about Arrow
>> dependency upgrades.
>>
>> The upgrade problem is not limited to Java bytecode compatibility. In our
>> experiments, newer Arrow versions failed at different layers. Arrow 18
>> requires a newer Java baseline than Calcite currently supports in its JDK 8
>> jobs. Arrow 17 and 16.1 still use Java 8 class files, but can hit Java
>> runtime API incompatibilities on JDK 8, such as `ByteBuffer.flip():
>> ByteBuffer`. Arrow 16.0 avoids that Java runtime issue, but exposed Gandiva
>> native / LLVM symbol issues on Linux CI.
>>
>> This means that as long as `arrow-gandiva` is required for the adapter's
>> correctness path, upgrading the Arrow Java vector layer also requires
>> validating the native Gandiva stack across all CI platforms. Even when
>> `arrow-vector` itself is usable, `arrow-gandiva` can still block the
>> upgrade.
>>
>> For that reason, I think the adapter should make projection/filter
>> correctness independent of Gandiva first. Once the Java correctness path is
>> in place, Arrow vector upgrades can be evaluated separately from Gandiva
>> native compatibility.
>>
>> The direction I have in mind is a pure Java correctness path for the Arrow
>> adapter:
>>
>> * read Arrow data with `ArrowFileReader`, `VectorSchemaRoot`, and
>> `ValueVector`;
>> * execute simple projections by reading selected vectors directly;
>> * execute the simple filters currently translated by `ArrowTranslator` with
>> a Java evaluator;
>> * leave expressions that are not pushed into the adapter to Calcite's
>> normal Enumerable / code generation path.
>>
>> With that model, Gandiva would no longer be required for correctness. A
>> staged migration could be:
>>
>> 1. Move no-filter simple projection away from Gandiva.
>> 2. Add Java evaluation for the simple filter subset currently supported by
>> `ArrowTranslator`.
>> 3. Validate that existing Arrow adapter tests pass without invoking
>> Gandiva.
>> 4. Remove `arrow-gandiva` from the adapter dependency set once the Java
>> path covers the current behavior.
>>
>> The tradeoff is that Gandiva may be faster for supported expressions. But
>> for this adapter, I think correctness, portability, and dependency
>> stability should come first. If acceleration is needed later, it can be
>> discussed separately.
>>
>> Does this direction make sense to the community? Are there current use
>> cases that depend on Gandiva pushdown strongly enough that we should keep
>> the native dependency?
>>
>> Thanks,
>> Cancai
>>
Re: Subject: [DISCUSS] Removing Gandiva from the Arrow adapter
Posted by Alessandro Solimando <al...@gmail.com>.
If it's not supported anymore it's already a good enough argument to
replace it.
The proposed plan looks reasonable to me.
Alessandro
On Thu, Jun 4, 2026, 09:45 jensen <cz...@163.com> wrote:
> I vaguely recall that Arrow's official website also discourages the use of
> Gandiva, and Gandiva is no longer maintained. If possible, I think we
> should remove this dependency.
>
>
>
> Best regards,
>
> Zhen
>
> ---- Replied Message ----
> | From | Cancai Cai<ca...@gmail.com> |
> | Date | 6/4/2026 14:20 |
> | To | <de...@calcite.apache.org> |
> | Subject | Subject: [DISCUSS] Removing Gandiva from the Arrow adapter |
> Hi all,
>
> I would like to discuss the future of Gandiva in Calcite's Arrow adapter.
> My preferred long-term direction is to remove the Gandiva dependency from
> the adapter.
>
> The current adapter uses Arrow Java to read Arrow data, but relies on
> Gandiva `Projector` and `Filter` for projection and filter execution.
> Gandiva is a native LLVM-based runtime, and the Java module is a wrapper
> around that native implementation. As a result, basic Arrow adapter queries
> depend on native libraries, LLVM compatibility, platform packaging, and JDK
> baseline details.
>
> This has become a practical maintenance problem when thinking about Arrow
> dependency upgrades.
>
> The upgrade problem is not limited to Java bytecode compatibility. In our
> experiments, newer Arrow versions failed at different layers. Arrow 18
> requires a newer Java baseline than Calcite currently supports in its JDK 8
> jobs. Arrow 17 and 16.1 still use Java 8 class files, but can hit Java
> runtime API incompatibilities on JDK 8, such as `ByteBuffer.flip():
> ByteBuffer`. Arrow 16.0 avoids that Java runtime issue, but exposed Gandiva
> native / LLVM symbol issues on Linux CI.
>
> This means that as long as `arrow-gandiva` is required for the adapter's
> correctness path, upgrading the Arrow Java vector layer also requires
> validating the native Gandiva stack across all CI platforms. Even when
> `arrow-vector` itself is usable, `arrow-gandiva` can still block the
> upgrade.
>
> For that reason, I think the adapter should make projection/filter
> correctness independent of Gandiva first. Once the Java correctness path is
> in place, Arrow vector upgrades can be evaluated separately from Gandiva
> native compatibility.
>
> The direction I have in mind is a pure Java correctness path for the Arrow
> adapter:
>
> * read Arrow data with `ArrowFileReader`, `VectorSchemaRoot`, and
> `ValueVector`;
> * execute simple projections by reading selected vectors directly;
> * execute the simple filters currently translated by `ArrowTranslator` with
> a Java evaluator;
> * leave expressions that are not pushed into the adapter to Calcite's
> normal Enumerable / code generation path.
>
> With that model, Gandiva would no longer be required for correctness. A
> staged migration could be:
>
> 1. Move no-filter simple projection away from Gandiva.
> 2. Add Java evaluation for the simple filter subset currently supported by
> `ArrowTranslator`.
> 3. Validate that existing Arrow adapter tests pass without invoking
> Gandiva.
> 4. Remove `arrow-gandiva` from the adapter dependency set once the Java
> path covers the current behavior.
>
> The tradeoff is that Gandiva may be faster for supported expressions. But
> for this adapter, I think correctness, portability, and dependency
> stability should come first. If acceleration is needed later, it can be
> discussed separately.
>
> Does this direction make sense to the community? Are there current use
> cases that depend on Gandiva pushdown strongly enough that we should keep
> the native dependency?
>
> Thanks,
> Cancai
>
Re: Subject: [DISCUSS] Removing Gandiva from the Arrow adapter
Posted by jensen <cz...@163.com>.
I vaguely recall that Arrow's official website also discourages the use of Gandiva, and Gandiva is no longer maintained. If possible, I think we should remove this dependency.
Best regards,
Zhen
---- Replied Message ----
| From | Cancai Cai<ca...@gmail.com> |
| Date | 6/4/2026 14:20 |
| To | <de...@calcite.apache.org> |
| Subject | Subject: [DISCUSS] Removing Gandiva from the Arrow adapter |
Hi all,
I would like to discuss the future of Gandiva in Calcite's Arrow adapter.
My preferred long-term direction is to remove the Gandiva dependency from
the adapter.
The current adapter uses Arrow Java to read Arrow data, but relies on
Gandiva `Projector` and `Filter` for projection and filter execution.
Gandiva is a native LLVM-based runtime, and the Java module is a wrapper
around that native implementation. As a result, basic Arrow adapter queries
depend on native libraries, LLVM compatibility, platform packaging, and JDK
baseline details.
This has become a practical maintenance problem when thinking about Arrow
dependency upgrades.
The upgrade problem is not limited to Java bytecode compatibility. In our
experiments, newer Arrow versions failed at different layers. Arrow 18
requires a newer Java baseline than Calcite currently supports in its JDK 8
jobs. Arrow 17 and 16.1 still use Java 8 class files, but can hit Java
runtime API incompatibilities on JDK 8, such as `ByteBuffer.flip():
ByteBuffer`. Arrow 16.0 avoids that Java runtime issue, but exposed Gandiva
native / LLVM symbol issues on Linux CI.
This means that as long as `arrow-gandiva` is required for the adapter's
correctness path, upgrading the Arrow Java vector layer also requires
validating the native Gandiva stack across all CI platforms. Even when
`arrow-vector` itself is usable, `arrow-gandiva` can still block the
upgrade.
For that reason, I think the adapter should make projection/filter
correctness independent of Gandiva first. Once the Java correctness path is
in place, Arrow vector upgrades can be evaluated separately from Gandiva
native compatibility.
The direction I have in mind is a pure Java correctness path for the Arrow
adapter:
* read Arrow data with `ArrowFileReader`, `VectorSchemaRoot`, and
`ValueVector`;
* execute simple projections by reading selected vectors directly;
* execute the simple filters currently translated by `ArrowTranslator` with
a Java evaluator;
* leave expressions that are not pushed into the adapter to Calcite's
normal Enumerable / code generation path.
With that model, Gandiva would no longer be required for correctness. A
staged migration could be:
1. Move no-filter simple projection away from Gandiva.
2. Add Java evaluation for the simple filter subset currently supported by
`ArrowTranslator`.
3. Validate that existing Arrow adapter tests pass without invoking Gandiva.
4. Remove `arrow-gandiva` from the adapter dependency set once the Java
path covers the current behavior.
The tradeoff is that Gandiva may be faster for supported expressions. But
for this adapter, I think correctness, portability, and dependency
stability should come first. If acceleration is needed later, it can be
discussed separately.
Does this direction make sense to the community? Are there current use
cases that depend on Gandiva pushdown strongly enough that we should keep
the native dependency?
Thanks,
Cancai