You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Jerome Haltom <jh...@alethic.solutions> on 2025/10/17 16:09:37 UTC

Calcite and !JDBC

Hello.

I am doing an interesting project (or so I think) using Calcite. I am attempting to write a .NET library that makes use of Calcite. The goal being to expose the Calcite operations one can normally do under JDBC in a nice .NET-like API surface using System.Data. So, a .NET class, Apache.Calcite.Data.CalciteConnection, which implements the .NET driver/connection model, but ultimately does the same thing the Calcite JDBC driver does. A nice clean .NET-like API surface that uses Calcite under the covers. I am doing this with IKVM (which is a JVM "implementation" that converts Java bytecode to .NET IL).

I have no problem fully using all the Calcite classes this way. Including the JDBC driver, etc. Everything works perfectly fine. So, my question is not about this translation layer, or this JVM implementation, or anything like that. I only mention it to provide context to my question.

My question is about how 'deep' into the core Calcite code the JDBC stuff extends.

I started this effort out by figuring I would mostly just need to rewrite a more appropriate version of org.apache.calcite.jdbc, that ultimately surfaced a .NET API but used all the same Calcite internals. But the more I dig into that, the more I find a few JDBC-isms that have snuck outside that package. For instance, org.apache.calcite.jdbc.CalciteScema is a big class. Doing quite a bit of work. And it's referenced by org.apache.calcite.prepare. And it's also accessed by the SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some stuff in calcite.schema.impl.ViewTable, materialize, etc.

Basically, JDBC extends deeper into Calcite than I had expected. So, I'm no longer just confronted with writing a new Connection-like implementation, but with new implementations of a number of other classes that seem pretty unrelated to JDBC itself.

So, then I think, okay. I may abandon that plan and instead preserve and not rewrite a class like CalciteSchema. But use the existing jdbc.CalciteSchema in my new CalciteConnection implementation. The namespaces/packages won't be as clear to users: they'll be creating jdbc.CalciteSchemas and adding them to Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should work.

So, how much of jdbc.* can I preserve, and how much is actually tied to JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not! CalcitePrepare has a few things in it, like CalciteSignature, which extends an Avatica class, Meta.Signature, and expects a list of AvaticaParameters. And this is very much JDBC-tied. But I need CalcitePrepare because of materialize/Lattice and all that.

So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I can't make use it since it relies on having a bunch of Avatica JDBC specific stuff exposed. Which a non-JDBC access method wouldn't have.

Any recommendations here? Ignore my .NET related context if you want, and pretend I'm writing a driver for some other driver framework that just has no support for JDBC-like stuff.



Re: Calcite and !JDBC

Posted by Stamatis Zampetakis <za...@gmail.com>.
Hi Jerome,

As you already observed JDBC concepts are not fully isolated in a
single package so if you want to mimic the capabilities of
jdbc:calcite you will probably have to touch/use many classes in core.

My first reaction when I saw the message was the same as what Damjan
proposed. If you want to do everything that a JDBC connection does
then it would be less work and probably more general to build a NET
wrapper over JDBC.

In some cases, it is indeed useful to have new APIs/drivers for
interacting with Calcite and I've done it for a few projects in the
past. The main difference though is that I was not aiming to replicate
what jdbc:connection offers but rather define APIs for specific DBMS
engines. Depending on exactly what the engine needs the difficulty and
the number of classes that you need to touch varies.

Best,
Stamatis

On Sun, Oct 19, 2025 at 2:43 PM Jerome Haltom <jh...@alethic.solutions> wrote:
>
> https://github.com/ikvmnet/calcite-dotnet/tree/main/src/Apache.Calcite.Adapter.Ado is the ADO backend I have working.
>
> This repository is early days however. That one project works, but the rest are in progress.
>
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Mark <ma...@gmail.com>
> Sent: Sunday, October 19, 2025 4:36:12 AM
> To: dev@calcite.apache.org <de...@calcite.apache.org>
> Cc: dev@calcite.apache.org <de...@calcite.apache.org>
> Subject: Re: Calcite and !JDBC
>
> Hi Jerome,
>
> Are you able to share your IKVM-based Calcite query engine repo? Had tried using IKVM sometime ago but wasn’t able to get a workable solution for querying custom data structures.
>
> Dotnet suffers from a dearth of efficient query engines. I found baby Kusto to be very accessible, and there is Grafana language server support for KQL, but it has no query optimizer. However KQL is very much locked down and effectively prevents contributing user defined functions, which is a bit of a blocker for me. Top tier Calcite support for dotnet, in any shape or form, would be amazing.
>
> -Mark
>
> > On 19 Oct 2025, at 15:04, Jerome Haltom <jh...@alethic.solutions> wrote:
> >
> > The backend is already finished. That was quite easy. Did not wrap JDBC, but did model my code on it
> >
> > https://github.com/ikvmnet/calcite-dotnet/tree/main/src/Apache.Calcite.Adapter.Ado
> >
> > Had to do some with on the dialect stuff though, since. Net DB driver's have no Metadata stuff.
> >
> > The front end is where in having problems, since I can't figure out what I'm going to have to replace.
> >
> > Get Outlook for Android<https://aka.ms/AAb9ysg>
> > ________________________________
> > From: Damjan Jovanovic <da...@apache.org>
> > Sent: Sunday, October 19, 2025 1:02:29 AM
> > To: dev@calcite.apache.org <de...@calcite.apache.org>
> > Subject: Re: Calcite and !JDBC
> >
> > To plan over .NET data structures, you might be able to develop a schema
> > adapter (https://calcite.apache.org/docs/adapter.html) that wraps those
> > .NET System.Data structures (Calcite -> .NET to Calcite schema adapter ->
> > .NET System.Data), which should be easier than trying to develop a complete
> > JDBC wrapper over .NET System.Data and then use it through the existing
> > JDBC schema adapter (Calcite -> JDBC schema adapter -> .NET to JDBC schema
> > adapter -> .NET System.Data). For an example you can look at the CSV schema
> > adapter (
> > https://github.com/apache/calcite/tree/main/example/csv/src/main/java/org/apache/calcite/adapter/csv
> > ).
> >
> > That's on the backend. The frontend to Calcite would still be JDBC, but
> > maybe https://github.com/chequer-io/JDBC.NET could help you wrap it into
> > .NET APIs?
> >
> > Damjan
> >
> >> On Sat, Oct 18, 2025 at 12:14 PM Jerome Haltom <jh...@alethic.solutions>
> >> wrote:
> >>
> >> Unfortunately that is completely insufficient for my needs. I'm using
> >> Calcite to plan over .NET data structures.
> >>
> >> Get Outlook for Android<https://aka.ms/AAb9ysg>
> >> ________________________________
> >> From: Istvan Toth <st...@cloudera.com.INVALID>
> >> Sent: Saturday, October 18, 2025 2:22:12 AM
> >> To: dev@calcite.apache.org <de...@calcite.apache.org>
> >> Subject: Re: Calcite and !JDBC
> >>
> >> The easiest way to do something like this is with Avatica.
> >>
> >> Avatica server exposes and proxies the JDBC operations over a simple
> >> protobuf RPC layer.
> >> The raw protobuf client can be trivially generated, and you can build the
> >> .Net API on top of that.
> >>
> >> For reference, the phoenix python connector does the same for the Python DB
> >> API:
> >> https://github.com/apache/phoenix-queryserver/tree/master/python-phoenixdb
> >>
> >> Here is an earler version of it without the phoenix-specic type mapping
> >> code:
> >> https://github.com/lalinsky/python-phoenixdb
> >>
> >> There is also an existing .NET client, but that does not include the
> >> mapping between the Avatica protocol and the native DB API.
> >> https://github.com/Azure/hdinsight-phoenix-sharp
> >>
> >> Istvan
> >>
> >>
> >> On Fri, Oct 17, 2025 at 6:10 PM Jerome Haltom <jh...@alethic.solutions>
> >> wrote:
> >>
> >>> Hello.
> >>>
> >>> I am doing an interesting project (or so I think) using Calcite. I am
> >>> attempting to write a .NET library that makes use of Calcite. The goal
> >>> being to expose the Calcite operations one can normally do under JDBC in
> >> a
> >>> nice .NET-like API surface using System.Data. So, a .NET class,
> >>> Apache.Calcite.Data.CalciteConnection, which implements the .NET
> >>> driver/connection model, but ultimately does the same thing the Calcite
> >>> JDBC driver does. A nice clean .NET-like API surface that uses Calcite
> >>> under the covers. I am doing this with IKVM (which is a JVM
> >>> "implementation" that converts Java bytecode to .NET IL).
> >>>
> >>> I have no problem fully using all the Calcite classes this way. Including
> >>> the JDBC driver, etc. Everything works perfectly fine. So, my question is
> >>> not about this translation layer, or this JVM implementation, or anything
> >>> like that. I only mention it to provide context to my question.
> >>>
> >>> My question is about how 'deep' into the core Calcite code the JDBC stuff
> >>> extends.
> >>>
> >>> I started this effort out by figuring I would mostly just need to rewrite
> >>> a more appropriate version of org.apache.calcite.jdbc, that ultimately
> >>> surfaced a .NET API but used all the same Calcite internals. But the
> >> more I
> >>> dig into that, the more I find a few JDBC-isms that have snuck outside
> >> that
> >>> package. For instance, org.apache.calcite.jdbc.CalciteScema is a big
> >> class.
> >>> Doing quite a bit of work. And it's referenced by
> >>> org.apache.calcite.prepare. And it's also accessed by the
> >>> SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some
> >>> stuff in calcite.schema.impl.ViewTable, materialize, etc.
> >>>
> >>> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm
> >>> no longer just confronted with writing a new Connection-like
> >>> implementation, but with new implementations of a number of other classes
> >>> that seem pretty unrelated to JDBC itself.
> >>>
> >>> So, then I think, okay. I may abandon that plan and instead preserve and
> >>> not rewrite a class like CalciteSchema. But use the existing
> >>> jdbc.CalciteSchema in my new CalciteConnection implementation. The
> >>> namespaces/packages won't be as clear to users: they'll be creating
> >>> jdbc.CalciteSchemas and adding them to
> >>> Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should
> >> work.
> >>>
> >>> So, how much of jdbc.* can I preserve, and how much is actually tied to
> >>> JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not!
> >>> CalcitePrepare has a few things in it, like CalciteSignature, which
> >> extends
> >>> an Avatica class, Meta.Signature, and expects a list of
> >> AvaticaParameters.
> >>> And this is very much JDBC-tied. But I need CalcitePrepare because of
> >>> materialize/Lattice and all that.
> >>>
> >>> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I
> >>> can't make use it since it relies on having a bunch of Avatica JDBC
> >>> specific stuff exposed. Which a non-JDBC access method wouldn't have.
> >>>
> >>> Any recommendations here? Ignore my .NET related context if you want, and
> >>> pretend I'm writing a driver for some other driver framework that just
> >> has
> >>> no support for JDBC-like stuff.
> >>>
> >>>
> >>>
> >>
> >> --
> >> *István Tóth* | Sr. Staff Software Engineer
> >> *Email*: stoty@cloudera.com
> >> cloudera.com <https://www.cloudera.com>
> >> [image: Cloudera] <https://www.cloudera.com/>
> >> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> >> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> >> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >> ------------------------------
> >> ------------------------------
> >>

Re: Calcite and !JDBC

Posted by Jerome Haltom <jh...@alethic.solutions>.
https://github.com/ikvmnet/calcite-dotnet/tree/main/src/Apache.Calcite.Adapter.Ado is the ADO backend I have working.

This repository is early days however. That one project works, but the rest are in progress.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Mark <ma...@gmail.com>
Sent: Sunday, October 19, 2025 4:36:12 AM
To: dev@calcite.apache.org <de...@calcite.apache.org>
Cc: dev@calcite.apache.org <de...@calcite.apache.org>
Subject: Re: Calcite and !JDBC

Hi Jerome,

Are you able to share your IKVM-based Calcite query engine repo? Had tried using IKVM sometime ago but wasn’t able to get a workable solution for querying custom data structures.

Dotnet suffers from a dearth of efficient query engines. I found baby Kusto to be very accessible, and there is Grafana language server support for KQL, but it has no query optimizer. However KQL is very much locked down and effectively prevents contributing user defined functions, which is a bit of a blocker for me. Top tier Calcite support for dotnet, in any shape or form, would be amazing.

-Mark

> On 19 Oct 2025, at 15:04, Jerome Haltom <jh...@alethic.solutions> wrote:
>
> The backend is already finished. That was quite easy. Did not wrap JDBC, but did model my code on it
>
> https://github.com/ikvmnet/calcite-dotnet/tree/main/src/Apache.Calcite.Adapter.Ado
>
> Had to do some with on the dialect stuff though, since. Net DB driver's have no Metadata stuff.
>
> The front end is where in having problems, since I can't figure out what I'm going to have to replace.
>
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Damjan Jovanovic <da...@apache.org>
> Sent: Sunday, October 19, 2025 1:02:29 AM
> To: dev@calcite.apache.org <de...@calcite.apache.org>
> Subject: Re: Calcite and !JDBC
>
> To plan over .NET data structures, you might be able to develop a schema
> adapter (https://calcite.apache.org/docs/adapter.html) that wraps those
> .NET System.Data structures (Calcite -> .NET to Calcite schema adapter ->
> .NET System.Data), which should be easier than trying to develop a complete
> JDBC wrapper over .NET System.Data and then use it through the existing
> JDBC schema adapter (Calcite -> JDBC schema adapter -> .NET to JDBC schema
> adapter -> .NET System.Data). For an example you can look at the CSV schema
> adapter (
> https://github.com/apache/calcite/tree/main/example/csv/src/main/java/org/apache/calcite/adapter/csv
> ).
>
> That's on the backend. The frontend to Calcite would still be JDBC, but
> maybe https://github.com/chequer-io/JDBC.NET could help you wrap it into
> .NET APIs?
>
> Damjan
>
>> On Sat, Oct 18, 2025 at 12:14 PM Jerome Haltom <jh...@alethic.solutions>
>> wrote:
>>
>> Unfortunately that is completely insufficient for my needs. I'm using
>> Calcite to plan over .NET data structures.
>>
>> Get Outlook for Android<https://aka.ms/AAb9ysg>
>> ________________________________
>> From: Istvan Toth <st...@cloudera.com.INVALID>
>> Sent: Saturday, October 18, 2025 2:22:12 AM
>> To: dev@calcite.apache.org <de...@calcite.apache.org>
>> Subject: Re: Calcite and !JDBC
>>
>> The easiest way to do something like this is with Avatica.
>>
>> Avatica server exposes and proxies the JDBC operations over a simple
>> protobuf RPC layer.
>> The raw protobuf client can be trivially generated, and you can build the
>> .Net API on top of that.
>>
>> For reference, the phoenix python connector does the same for the Python DB
>> API:
>> https://github.com/apache/phoenix-queryserver/tree/master/python-phoenixdb
>>
>> Here is an earler version of it without the phoenix-specic type mapping
>> code:
>> https://github.com/lalinsky/python-phoenixdb
>>
>> There is also an existing .NET client, but that does not include the
>> mapping between the Avatica protocol and the native DB API.
>> https://github.com/Azure/hdinsight-phoenix-sharp
>>
>> Istvan
>>
>>
>> On Fri, Oct 17, 2025 at 6:10 PM Jerome Haltom <jh...@alethic.solutions>
>> wrote:
>>
>>> Hello.
>>>
>>> I am doing an interesting project (or so I think) using Calcite. I am
>>> attempting to write a .NET library that makes use of Calcite. The goal
>>> being to expose the Calcite operations one can normally do under JDBC in
>> a
>>> nice .NET-like API surface using System.Data. So, a .NET class,
>>> Apache.Calcite.Data.CalciteConnection, which implements the .NET
>>> driver/connection model, but ultimately does the same thing the Calcite
>>> JDBC driver does. A nice clean .NET-like API surface that uses Calcite
>>> under the covers. I am doing this with IKVM (which is a JVM
>>> "implementation" that converts Java bytecode to .NET IL).
>>>
>>> I have no problem fully using all the Calcite classes this way. Including
>>> the JDBC driver, etc. Everything works perfectly fine. So, my question is
>>> not about this translation layer, or this JVM implementation, or anything
>>> like that. I only mention it to provide context to my question.
>>>
>>> My question is about how 'deep' into the core Calcite code the JDBC stuff
>>> extends.
>>>
>>> I started this effort out by figuring I would mostly just need to rewrite
>>> a more appropriate version of org.apache.calcite.jdbc, that ultimately
>>> surfaced a .NET API but used all the same Calcite internals. But the
>> more I
>>> dig into that, the more I find a few JDBC-isms that have snuck outside
>> that
>>> package. For instance, org.apache.calcite.jdbc.CalciteScema is a big
>> class.
>>> Doing quite a bit of work. And it's referenced by
>>> org.apache.calcite.prepare. And it's also accessed by the
>>> SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some
>>> stuff in calcite.schema.impl.ViewTable, materialize, etc.
>>>
>>> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm
>>> no longer just confronted with writing a new Connection-like
>>> implementation, but with new implementations of a number of other classes
>>> that seem pretty unrelated to JDBC itself.
>>>
>>> So, then I think, okay. I may abandon that plan and instead preserve and
>>> not rewrite a class like CalciteSchema. But use the existing
>>> jdbc.CalciteSchema in my new CalciteConnection implementation. The
>>> namespaces/packages won't be as clear to users: they'll be creating
>>> jdbc.CalciteSchemas and adding them to
>>> Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should
>> work.
>>>
>>> So, how much of jdbc.* can I preserve, and how much is actually tied to
>>> JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not!
>>> CalcitePrepare has a few things in it, like CalciteSignature, which
>> extends
>>> an Avatica class, Meta.Signature, and expects a list of
>> AvaticaParameters.
>>> And this is very much JDBC-tied. But I need CalcitePrepare because of
>>> materialize/Lattice and all that.
>>>
>>> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I
>>> can't make use it since it relies on having a bunch of Avatica JDBC
>>> specific stuff exposed. Which a non-JDBC access method wouldn't have.
>>>
>>> Any recommendations here? Ignore my .NET related context if you want, and
>>> pretend I'm writing a driver for some other driver framework that just
>> has
>>> no support for JDBC-like stuff.
>>>
>>>
>>>
>>
>> --
>> *István Tóth* | Sr. Staff Software Engineer
>> *Email*: stoty@cloudera.com
>> cloudera.com <https://www.cloudera.com>
>> [image: Cloudera] <https://www.cloudera.com/>
>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
>> on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> ------------------------------
>> ------------------------------
>>

Re: Calcite and !JDBC

Posted by Mark <ma...@gmail.com>.
Hi Jerome,

Are you able to share your IKVM-based Calcite query engine repo? Had tried using IKVM sometime ago but wasn’t able to get a workable solution for querying custom data structures.

Dotnet suffers from a dearth of efficient query engines. I found baby Kusto to be very accessible, and there is Grafana language server support for KQL, but it has no query optimizer. However KQL is very much locked down and effectively prevents contributing user defined functions, which is a bit of a blocker for me. Top tier Calcite support for dotnet, in any shape or form, would be amazing.

-Mark 

> On 19 Oct 2025, at 15:04, Jerome Haltom <jh...@alethic.solutions> wrote:
> 
> The backend is already finished. That was quite easy. Did not wrap JDBC, but did model my code on it
> 
> https://github.com/ikvmnet/calcite-dotnet/tree/main/src/Apache.Calcite.Adapter.Ado
> 
> Had to do some with on the dialect stuff though, since. Net DB driver's have no Metadata stuff.
> 
> The front end is where in having problems, since I can't figure out what I'm going to have to replace.
> 
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Damjan Jovanovic <da...@apache.org>
> Sent: Sunday, October 19, 2025 1:02:29 AM
> To: dev@calcite.apache.org <de...@calcite.apache.org>
> Subject: Re: Calcite and !JDBC
> 
> To plan over .NET data structures, you might be able to develop a schema
> adapter (https://calcite.apache.org/docs/adapter.html) that wraps those
> .NET System.Data structures (Calcite -> .NET to Calcite schema adapter ->
> .NET System.Data), which should be easier than trying to develop a complete
> JDBC wrapper over .NET System.Data and then use it through the existing
> JDBC schema adapter (Calcite -> JDBC schema adapter -> .NET to JDBC schema
> adapter -> .NET System.Data). For an example you can look at the CSV schema
> adapter (
> https://github.com/apache/calcite/tree/main/example/csv/src/main/java/org/apache/calcite/adapter/csv
> ).
> 
> That's on the backend. The frontend to Calcite would still be JDBC, but
> maybe https://github.com/chequer-io/JDBC.NET could help you wrap it into
> .NET APIs?
> 
> Damjan
> 
>> On Sat, Oct 18, 2025 at 12:14 PM Jerome Haltom <jh...@alethic.solutions>
>> wrote:
>> 
>> Unfortunately that is completely insufficient for my needs. I'm using
>> Calcite to plan over .NET data structures.
>> 
>> Get Outlook for Android<https://aka.ms/AAb9ysg>
>> ________________________________
>> From: Istvan Toth <st...@cloudera.com.INVALID>
>> Sent: Saturday, October 18, 2025 2:22:12 AM
>> To: dev@calcite.apache.org <de...@calcite.apache.org>
>> Subject: Re: Calcite and !JDBC
>> 
>> The easiest way to do something like this is with Avatica.
>> 
>> Avatica server exposes and proxies the JDBC operations over a simple
>> protobuf RPC layer.
>> The raw protobuf client can be trivially generated, and you can build the
>> .Net API on top of that.
>> 
>> For reference, the phoenix python connector does the same for the Python DB
>> API:
>> https://github.com/apache/phoenix-queryserver/tree/master/python-phoenixdb
>> 
>> Here is an earler version of it without the phoenix-specic type mapping
>> code:
>> https://github.com/lalinsky/python-phoenixdb
>> 
>> There is also an existing .NET client, but that does not include the
>> mapping between the Avatica protocol and the native DB API.
>> https://github.com/Azure/hdinsight-phoenix-sharp
>> 
>> Istvan
>> 
>> 
>> On Fri, Oct 17, 2025 at 6:10 PM Jerome Haltom <jh...@alethic.solutions>
>> wrote:
>> 
>>> Hello.
>>> 
>>> I am doing an interesting project (or so I think) using Calcite. I am
>>> attempting to write a .NET library that makes use of Calcite. The goal
>>> being to expose the Calcite operations one can normally do under JDBC in
>> a
>>> nice .NET-like API surface using System.Data. So, a .NET class,
>>> Apache.Calcite.Data.CalciteConnection, which implements the .NET
>>> driver/connection model, but ultimately does the same thing the Calcite
>>> JDBC driver does. A nice clean .NET-like API surface that uses Calcite
>>> under the covers. I am doing this with IKVM (which is a JVM
>>> "implementation" that converts Java bytecode to .NET IL).
>>> 
>>> I have no problem fully using all the Calcite classes this way. Including
>>> the JDBC driver, etc. Everything works perfectly fine. So, my question is
>>> not about this translation layer, or this JVM implementation, or anything
>>> like that. I only mention it to provide context to my question.
>>> 
>>> My question is about how 'deep' into the core Calcite code the JDBC stuff
>>> extends.
>>> 
>>> I started this effort out by figuring I would mostly just need to rewrite
>>> a more appropriate version of org.apache.calcite.jdbc, that ultimately
>>> surfaced a .NET API but used all the same Calcite internals. But the
>> more I
>>> dig into that, the more I find a few JDBC-isms that have snuck outside
>> that
>>> package. For instance, org.apache.calcite.jdbc.CalciteScema is a big
>> class.
>>> Doing quite a bit of work. And it's referenced by
>>> org.apache.calcite.prepare. And it's also accessed by the
>>> SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some
>>> stuff in calcite.schema.impl.ViewTable, materialize, etc.
>>> 
>>> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm
>>> no longer just confronted with writing a new Connection-like
>>> implementation, but with new implementations of a number of other classes
>>> that seem pretty unrelated to JDBC itself.
>>> 
>>> So, then I think, okay. I may abandon that plan and instead preserve and
>>> not rewrite a class like CalciteSchema. But use the existing
>>> jdbc.CalciteSchema in my new CalciteConnection implementation. The
>>> namespaces/packages won't be as clear to users: they'll be creating
>>> jdbc.CalciteSchemas and adding them to
>>> Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should
>> work.
>>> 
>>> So, how much of jdbc.* can I preserve, and how much is actually tied to
>>> JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not!
>>> CalcitePrepare has a few things in it, like CalciteSignature, which
>> extends
>>> an Avatica class, Meta.Signature, and expects a list of
>> AvaticaParameters.
>>> And this is very much JDBC-tied. But I need CalcitePrepare because of
>>> materialize/Lattice and all that.
>>> 
>>> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I
>>> can't make use it since it relies on having a bunch of Avatica JDBC
>>> specific stuff exposed. Which a non-JDBC access method wouldn't have.
>>> 
>>> Any recommendations here? Ignore my .NET related context if you want, and
>>> pretend I'm writing a driver for some other driver framework that just
>> has
>>> no support for JDBC-like stuff.
>>> 
>>> 
>>> 
>> 
>> --
>> *István Tóth* | Sr. Staff Software Engineer
>> *Email*: stoty@cloudera.com
>> cloudera.com <https://www.cloudera.com>
>> [image: Cloudera] <https://www.cloudera.com/>
>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
>> on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> ------------------------------
>> ------------------------------
>> 

Re: Calcite and !JDBC

Posted by Jerome Haltom <jh...@alethic.solutions>.
The backend is already finished. That was quite easy. Did not wrap JDBC, but did model my code on it

https://github.com/ikvmnet/calcite-dotnet/tree/main/src/Apache.Calcite.Adapter.Ado

Had to do some with on the dialect stuff though, since. Net DB driver's have no Metadata stuff.

The front end is where in having problems, since I can't figure out what I'm going to have to replace.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Damjan Jovanovic <da...@apache.org>
Sent: Sunday, October 19, 2025 1:02:29 AM
To: dev@calcite.apache.org <de...@calcite.apache.org>
Subject: Re: Calcite and !JDBC

To plan over .NET data structures, you might be able to develop a schema
adapter (https://calcite.apache.org/docs/adapter.html) that wraps those
.NET System.Data structures (Calcite -> .NET to Calcite schema adapter ->
.NET System.Data), which should be easier than trying to develop a complete
JDBC wrapper over .NET System.Data and then use it through the existing
JDBC schema adapter (Calcite -> JDBC schema adapter -> .NET to JDBC schema
adapter -> .NET System.Data). For an example you can look at the CSV schema
adapter (
https://github.com/apache/calcite/tree/main/example/csv/src/main/java/org/apache/calcite/adapter/csv
).

That's on the backend. The frontend to Calcite would still be JDBC, but
maybe https://github.com/chequer-io/JDBC.NET could help you wrap it into
.NET APIs?

Damjan

On Sat, Oct 18, 2025 at 12:14 PM Jerome Haltom <jh...@alethic.solutions>
wrote:

> Unfortunately that is completely insufficient for my needs. I'm using
> Calcite to plan over .NET data structures.
>
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Istvan Toth <st...@cloudera.com.INVALID>
> Sent: Saturday, October 18, 2025 2:22:12 AM
> To: dev@calcite.apache.org <de...@calcite.apache.org>
> Subject: Re: Calcite and !JDBC
>
> The easiest way to do something like this is with Avatica.
>
> Avatica server exposes and proxies the JDBC operations over a simple
> protobuf RPC layer.
> The raw protobuf client can be trivially generated, and you can build the
> .Net API on top of that.
>
> For reference, the phoenix python connector does the same for the Python DB
> API:
> https://github.com/apache/phoenix-queryserver/tree/master/python-phoenixdb
>
> Here is an earler version of it without the phoenix-specic type mapping
> code:
> https://github.com/lalinsky/python-phoenixdb
>
> There is also an existing .NET client, but that does not include the
> mapping between the Avatica protocol and the native DB API.
> https://github.com/Azure/hdinsight-phoenix-sharp
>
> Istvan
>
>
> On Fri, Oct 17, 2025 at 6:10 PM Jerome Haltom <jh...@alethic.solutions>
> wrote:
>
> > Hello.
> >
> > I am doing an interesting project (or so I think) using Calcite. I am
> > attempting to write a .NET library that makes use of Calcite. The goal
> > being to expose the Calcite operations one can normally do under JDBC in
> a
> > nice .NET-like API surface using System.Data. So, a .NET class,
> > Apache.Calcite.Data.CalciteConnection, which implements the .NET
> > driver/connection model, but ultimately does the same thing the Calcite
> > JDBC driver does. A nice clean .NET-like API surface that uses Calcite
> > under the covers. I am doing this with IKVM (which is a JVM
> > "implementation" that converts Java bytecode to .NET IL).
> >
> > I have no problem fully using all the Calcite classes this way. Including
> > the JDBC driver, etc. Everything works perfectly fine. So, my question is
> > not about this translation layer, or this JVM implementation, or anything
> > like that. I only mention it to provide context to my question.
> >
> > My question is about how 'deep' into the core Calcite code the JDBC stuff
> > extends.
> >
> > I started this effort out by figuring I would mostly just need to rewrite
> > a more appropriate version of org.apache.calcite.jdbc, that ultimately
> > surfaced a .NET API but used all the same Calcite internals. But the
> more I
> > dig into that, the more I find a few JDBC-isms that have snuck outside
> that
> > package. For instance, org.apache.calcite.jdbc.CalciteScema is a big
> class.
> > Doing quite a bit of work. And it's referenced by
> > org.apache.calcite.prepare. And it's also accessed by the
> > SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some
> > stuff in calcite.schema.impl.ViewTable, materialize, etc.
> >
> > Basically, JDBC extends deeper into Calcite than I had expected. So, I'm
> > no longer just confronted with writing a new Connection-like
> > implementation, but with new implementations of a number of other classes
> > that seem pretty unrelated to JDBC itself.
> >
> > So, then I think, okay. I may abandon that plan and instead preserve and
> > not rewrite a class like CalciteSchema. But use the existing
> > jdbc.CalciteSchema in my new CalciteConnection implementation. The
> > namespaces/packages won't be as clear to users: they'll be creating
> > jdbc.CalciteSchemas and adding them to
> > Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should
> work.
> >
> > So, how much of jdbc.* can I preserve, and how much is actually tied to
> > JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not!
> > CalcitePrepare has a few things in it, like CalciteSignature, which
> extends
> > an Avatica class, Meta.Signature, and expects a list of
> AvaticaParameters.
> > And this is very much JDBC-tied. But I need CalcitePrepare because of
> > materialize/Lattice and all that.
> >
> > So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I
> > can't make use it since it relies on having a bunch of Avatica JDBC
> > specific stuff exposed. Which a non-JDBC access method wouldn't have.
> >
> > Any recommendations here? Ignore my .NET related context if you want, and
> > pretend I'm writing a driver for some other driver framework that just
> has
> > no support for JDBC-like stuff.
> >
> >
> >
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>

Re: Calcite and !JDBC

Posted by Damjan Jovanovic <da...@apache.org>.
To plan over .NET data structures, you might be able to develop a schema
adapter (https://calcite.apache.org/docs/adapter.html) that wraps those
.NET System.Data structures (Calcite -> .NET to Calcite schema adapter ->
.NET System.Data), which should be easier than trying to develop a complete
JDBC wrapper over .NET System.Data and then use it through the existing
JDBC schema adapter (Calcite -> JDBC schema adapter -> .NET to JDBC schema
adapter -> .NET System.Data). For an example you can look at the CSV schema
adapter (
https://github.com/apache/calcite/tree/main/example/csv/src/main/java/org/apache/calcite/adapter/csv
).

That's on the backend. The frontend to Calcite would still be JDBC, but
maybe https://github.com/chequer-io/JDBC.NET could help you wrap it into
.NET APIs?

Damjan

On Sat, Oct 18, 2025 at 12:14 PM Jerome Haltom <jh...@alethic.solutions>
wrote:

> Unfortunately that is completely insufficient for my needs. I'm using
> Calcite to plan over .NET data structures.
>
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Istvan Toth <st...@cloudera.com.INVALID>
> Sent: Saturday, October 18, 2025 2:22:12 AM
> To: dev@calcite.apache.org <de...@calcite.apache.org>
> Subject: Re: Calcite and !JDBC
>
> The easiest way to do something like this is with Avatica.
>
> Avatica server exposes and proxies the JDBC operations over a simple
> protobuf RPC layer.
> The raw protobuf client can be trivially generated, and you can build the
> .Net API on top of that.
>
> For reference, the phoenix python connector does the same for the Python DB
> API:
> https://github.com/apache/phoenix-queryserver/tree/master/python-phoenixdb
>
> Here is an earler version of it without the phoenix-specic type mapping
> code:
> https://github.com/lalinsky/python-phoenixdb
>
> There is also an existing .NET client, but that does not include the
> mapping between the Avatica protocol and the native DB API.
> https://github.com/Azure/hdinsight-phoenix-sharp
>
> Istvan
>
>
> On Fri, Oct 17, 2025 at 6:10 PM Jerome Haltom <jh...@alethic.solutions>
> wrote:
>
> > Hello.
> >
> > I am doing an interesting project (or so I think) using Calcite. I am
> > attempting to write a .NET library that makes use of Calcite. The goal
> > being to expose the Calcite operations one can normally do under JDBC in
> a
> > nice .NET-like API surface using System.Data. So, a .NET class,
> > Apache.Calcite.Data.CalciteConnection, which implements the .NET
> > driver/connection model, but ultimately does the same thing the Calcite
> > JDBC driver does. A nice clean .NET-like API surface that uses Calcite
> > under the covers. I am doing this with IKVM (which is a JVM
> > "implementation" that converts Java bytecode to .NET IL).
> >
> > I have no problem fully using all the Calcite classes this way. Including
> > the JDBC driver, etc. Everything works perfectly fine. So, my question is
> > not about this translation layer, or this JVM implementation, or anything
> > like that. I only mention it to provide context to my question.
> >
> > My question is about how 'deep' into the core Calcite code the JDBC stuff
> > extends.
> >
> > I started this effort out by figuring I would mostly just need to rewrite
> > a more appropriate version of org.apache.calcite.jdbc, that ultimately
> > surfaced a .NET API but used all the same Calcite internals. But the
> more I
> > dig into that, the more I find a few JDBC-isms that have snuck outside
> that
> > package. For instance, org.apache.calcite.jdbc.CalciteScema is a big
> class.
> > Doing quite a bit of work. And it's referenced by
> > org.apache.calcite.prepare. And it's also accessed by the
> > SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some
> > stuff in calcite.schema.impl.ViewTable, materialize, etc.
> >
> > Basically, JDBC extends deeper into Calcite than I had expected. So, I'm
> > no longer just confronted with writing a new Connection-like
> > implementation, but with new implementations of a number of other classes
> > that seem pretty unrelated to JDBC itself.
> >
> > So, then I think, okay. I may abandon that plan and instead preserve and
> > not rewrite a class like CalciteSchema. But use the existing
> > jdbc.CalciteSchema in my new CalciteConnection implementation. The
> > namespaces/packages won't be as clear to users: they'll be creating
> > jdbc.CalciteSchemas and adding them to
> > Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should
> work.
> >
> > So, how much of jdbc.* can I preserve, and how much is actually tied to
> > JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not!
> > CalcitePrepare has a few things in it, like CalciteSignature, which
> extends
> > an Avatica class, Meta.Signature, and expects a list of
> AvaticaParameters.
> > And this is very much JDBC-tied. But I need CalcitePrepare because of
> > materialize/Lattice and all that.
> >
> > So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I
> > can't make use it since it relies on having a bunch of Avatica JDBC
> > specific stuff exposed. Which a non-JDBC access method wouldn't have.
> >
> > Any recommendations here? Ignore my .NET related context if you want, and
> > pretend I'm writing a driver for some other driver framework that just
> has
> > no support for JDBC-like stuff.
> >
> >
> >
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: stoty@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
> ------------------------------
>

Re: Calcite and !JDBC

Posted by Jerome Haltom <jh...@alethic.solutions>.
Unfortunately that is completely insufficient for my needs. I'm using Calcite to plan over .NET data structures.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Istvan Toth <st...@cloudera.com.INVALID>
Sent: Saturday, October 18, 2025 2:22:12 AM
To: dev@calcite.apache.org <de...@calcite.apache.org>
Subject: Re: Calcite and !JDBC

The easiest way to do something like this is with Avatica.

Avatica server exposes and proxies the JDBC operations over a simple
protobuf RPC layer.
The raw protobuf client can be trivially generated, and you can build the
.Net API on top of that.

For reference, the phoenix python connector does the same for the Python DB
API:
https://github.com/apache/phoenix-queryserver/tree/master/python-phoenixdb

Here is an earler version of it without the phoenix-specic type mapping
code:
https://github.com/lalinsky/python-phoenixdb

There is also an existing .NET client, but that does not include the
mapping between the Avatica protocol and the native DB API.
https://github.com/Azure/hdinsight-phoenix-sharp

Istvan


On Fri, Oct 17, 2025 at 6:10 PM Jerome Haltom <jh...@alethic.solutions>
wrote:

> Hello.
>
> I am doing an interesting project (or so I think) using Calcite. I am
> attempting to write a .NET library that makes use of Calcite. The goal
> being to expose the Calcite operations one can normally do under JDBC in a
> nice .NET-like API surface using System.Data. So, a .NET class,
> Apache.Calcite.Data.CalciteConnection, which implements the .NET
> driver/connection model, but ultimately does the same thing the Calcite
> JDBC driver does. A nice clean .NET-like API surface that uses Calcite
> under the covers. I am doing this with IKVM (which is a JVM
> "implementation" that converts Java bytecode to .NET IL).
>
> I have no problem fully using all the Calcite classes this way. Including
> the JDBC driver, etc. Everything works perfectly fine. So, my question is
> not about this translation layer, or this JVM implementation, or anything
> like that. I only mention it to provide context to my question.
>
> My question is about how 'deep' into the core Calcite code the JDBC stuff
> extends.
>
> I started this effort out by figuring I would mostly just need to rewrite
> a more appropriate version of org.apache.calcite.jdbc, that ultimately
> surfaced a .NET API but used all the same Calcite internals. But the more I
> dig into that, the more I find a few JDBC-isms that have snuck outside that
> package. For instance, org.apache.calcite.jdbc.CalciteScema is a big class.
> Doing quite a bit of work. And it's referenced by
> org.apache.calcite.prepare. And it's also accessed by the
> SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some
> stuff in calcite.schema.impl.ViewTable, materialize, etc.
>
> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm
> no longer just confronted with writing a new Connection-like
> implementation, but with new implementations of a number of other classes
> that seem pretty unrelated to JDBC itself.
>
> So, then I think, okay. I may abandon that plan and instead preserve and
> not rewrite a class like CalciteSchema. But use the existing
> jdbc.CalciteSchema in my new CalciteConnection implementation. The
> namespaces/packages won't be as clear to users: they'll be creating
> jdbc.CalciteSchemas and adding them to
> Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should work.
>
> So, how much of jdbc.* can I preserve, and how much is actually tied to
> JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not!
> CalcitePrepare has a few things in it, like CalciteSignature, which extends
> an Avatica class, Meta.Signature, and expects a list of AvaticaParameters.
> And this is very much JDBC-tied. But I need CalcitePrepare because of
> materialize/Lattice and all that.
>
> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I
> can't make use it since it relies on having a bunch of Avatica JDBC
> specific stuff exposed. Which a non-JDBC access method wouldn't have.
>
> Any recommendations here? Ignore my .NET related context if you want, and
> pretend I'm writing a driver for some other driver framework that just has
> no support for JDBC-like stuff.
>
>
>

--
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------

Re: Calcite and !JDBC

Posted by Istvan Toth <st...@cloudera.com.INVALID>.
The easiest way to do something like this is with Avatica.

Avatica server exposes and proxies the JDBC operations over a simple
protobuf RPC layer.
The raw protobuf client can be trivially generated, and you can build the
.Net API on top of that.

For reference, the phoenix python connector does the same for the Python DB
API:
https://github.com/apache/phoenix-queryserver/tree/master/python-phoenixdb

Here is an earler version of it without the phoenix-specic type mapping
code:
https://github.com/lalinsky/python-phoenixdb

There is also an existing .NET client, but that does not include the
mapping between the Avatica protocol and the native DB API.
https://github.com/Azure/hdinsight-phoenix-sharp

Istvan


On Fri, Oct 17, 2025 at 6:10 PM Jerome Haltom <jh...@alethic.solutions>
wrote:

> Hello.
>
> I am doing an interesting project (or so I think) using Calcite. I am
> attempting to write a .NET library that makes use of Calcite. The goal
> being to expose the Calcite operations one can normally do under JDBC in a
> nice .NET-like API surface using System.Data. So, a .NET class,
> Apache.Calcite.Data.CalciteConnection, which implements the .NET
> driver/connection model, but ultimately does the same thing the Calcite
> JDBC driver does. A nice clean .NET-like API surface that uses Calcite
> under the covers. I am doing this with IKVM (which is a JVM
> "implementation" that converts Java bytecode to .NET IL).
>
> I have no problem fully using all the Calcite classes this way. Including
> the JDBC driver, etc. Everything works perfectly fine. So, my question is
> not about this translation layer, or this JVM implementation, or anything
> like that. I only mention it to provide context to my question.
>
> My question is about how 'deep' into the core Calcite code the JDBC stuff
> extends.
>
> I started this effort out by figuring I would mostly just need to rewrite
> a more appropriate version of org.apache.calcite.jdbc, that ultimately
> surfaced a .NET API but used all the same Calcite internals. But the more I
> dig into that, the more I find a few JDBC-isms that have snuck outside that
> package. For instance, org.apache.calcite.jdbc.CalciteScema is a big class.
> Doing quite a bit of work. And it's referenced by
> org.apache.calcite.prepare. And it's also accessed by the
> SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some
> stuff in calcite.schema.impl.ViewTable, materialize, etc.
>
> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm
> no longer just confronted with writing a new Connection-like
> implementation, but with new implementations of a number of other classes
> that seem pretty unrelated to JDBC itself.
>
> So, then I think, okay. I may abandon that plan and instead preserve and
> not rewrite a class like CalciteSchema. But use the existing
> jdbc.CalciteSchema in my new CalciteConnection implementation. The
> namespaces/packages won't be as clear to users: they'll be creating
> jdbc.CalciteSchemas and adding them to
> Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should work.
>
> So, how much of jdbc.* can I preserve, and how much is actually tied to
> JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not!
> CalcitePrepare has a few things in it, like CalciteSignature, which extends
> an Avatica class, Meta.Signature, and expects a list of AvaticaParameters.
> And this is very much JDBC-tied. But I need CalcitePrepare because of
> materialize/Lattice and all that.
>
> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I
> can't make use it since it relies on having a bunch of Avatica JDBC
> specific stuff exposed. Which a non-JDBC access method wouldn't have.
>
> Any recommendations here? Ignore my .NET related context if you want, and
> pretend I'm writing a driver for some other driver framework that just has
> no support for JDBC-like stuff.
>
>
>

-- 
*István Tóth* | Sr. Staff Software Engineer
*Email*: stoty@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
------------------------------

Re: Calcite and !JDBC

Posted by Jerome Haltom <jh...@alethic.solutions>.
Yes.... I know that...

Did you see the classes I asked about in my initial post?

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Julian Hyde <jh...@gmail.com>
Sent: Saturday, October 18, 2025 9:26:54 PM
To: dev@calcite.apache.org <de...@calcite.apache.org>
Subject: Re: Calcite and !JDBC

Well of course Calcite is connected to JDBC. It needs an INTEGER type that other dialects recognize as an INTEGER type. So it uses java.sql.Types.INTEGER, whose value is 4.

In the .NET world you have DataTypeEnum.adInteger, whose value is 3, and maps onto ODBC’s DBTYPE_I4.

The concepts are largely the same, but the values are different. You have to do that mapping. IKVM can’t do it, because the .NET concepts don’t exist in the Java code it is translating.

> On Oct 18, 2025, at 7:16 PM, Jerome Haltom <jh...@alethic.solutions> wrote:
>
> IKVM does 100%. But it is not a code translator. Byte code to IL.
>
> We pass the full JDK test suite, for JDK8 anyways.
>
> So you can kind of forget I'm using it, and pretend I'm just asking about Java, but I've got a custom driver framework that isn't JDBC, so I'm trying to do what Calcites JDBC driver does without JDBC.
>
> The question is where in Calcite does the JDBC end and the Calcite begin, if you get my drift.
>
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Julian Hyde <jh...@gmail.com>
> Sent: Saturday, October 18, 2025 9:04:56 PM
> To: dev@calcite.apache.org <de...@calcite.apache.org>
> Subject: Re: Calcite and !JDBC
>
> It’s no accident that the comment at the top of CalcitePrepareImpl is “Shit just got real”. It’s the coordinator of the planning process, so needs to deal with both expression algebra and dynamics (generating a statement id, so that the statement can hook into resource and lifecycle management). It’s hard to cut this Gordian knot.
>
> Can you mock the JDBC bits, or map them onto .NET concepts? (They’re not too dissimilar… both evolved from ODBC.)
>
> If I were taking on this project, and using an automated translation tool like IKVM, I would make a clone of the Calcite Java code and refactor it a bit. I don’t know IKVM but I suspect that it can do 90% or 99% but struggles with the last 1%. Once you’ve discovered which refactorings made it work, you’ve discovered which are the right cuts in the Gordian knot, and can perhaps contribute those back to Calcite.
>
> Julian
>
>
>> On Oct 17, 2025, at 9:09 AM, Jerome Haltom <jh...@alethic.solutions> wrote:
>>
>> Hello.
>>
>> I am doing an interesting project (or so I think) using Calcite. I am attempting to write a .NET library that makes use of Calcite. The goal being to expose the Calcite operations one can normally do under JDBC in a nice .NET-like API surface using System.Data. So, a .NET class, Apache.Calcite.Data.CalciteConnection, which implements the .NET driver/connection model, but ultimately does the same thing the Calcite JDBC driver does. A nice clean .NET-like API surface that uses Calcite under the covers. I am doing this with IKVM (which is a JVM "implementation" that converts Java bytecode to .NET IL).
>>
>> I have no problem fully using all the Calcite classes this way. Including the JDBC driver, etc. Everything works perfectly fine. So, my question is not about this translation layer, or this JVM implementation, or anything like that. I only mention it to provide context to my question.
>>
>> My question is about how 'deep' into the core Calcite code the JDBC stuff extends.
>>
>> I started this effort out by figuring I would mostly just need to rewrite a more appropriate version of org.apache.calcite.jdbc, that ultimately surfaced a .NET API but used all the same Calcite internals. But the more I dig into that, the more I find a few JDBC-isms that have snuck outside that package. For instance, org.apache.calcite.jdbc.CalciteScema is a big class. Doing quite a bit of work. And it's referenced by org.apache.calcite.prepare. And it's also accessed by the SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some stuff in calcite.schema.impl.ViewTable, materialize, etc.
>>
>> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm no longer just confronted with writing a new Connection-like implementation, but with new implementations of a number of other classes that seem pretty unrelated to JDBC itself.
>>
>> So, then I think, okay. I may abandon that plan and instead preserve and not rewrite a class like CalciteSchema. But use the existing jdbc.CalciteSchema in my new CalciteConnection implementation. The namespaces/packages won't be as clear to users: they'll be creating jdbc.CalciteSchemas and adding them to Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should work.
>>
>> So, how much of jdbc.* can I preserve, and how much is actually tied to JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not! CalcitePrepare has a few things in it, like CalciteSignature, which extends an Avatica class, Meta.Signature, and expects a list of AvaticaParameters. And this is very much JDBC-tied. But I need CalcitePrepare because of materialize/Lattice and all that.
>>
>> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I can't make use it since it relies on having a bunch of Avatica JDBC specific stuff exposed. Which a non-JDBC access method wouldn't have.
>>
>> Any recommendations here? Ignore my .NET related context if you want, and pretend I'm writing a driver for some other driver framework that just has no support for JDBC-like stuff.
>>
>>
>


Re: Calcite and !JDBC

Posted by Julian Hyde <jh...@gmail.com>.
Well of course Calcite is connected to JDBC. It needs an INTEGER type that other dialects recognize as an INTEGER type. So it uses java.sql.Types.INTEGER, whose value is 4. 

In the .NET world you have DataTypeEnum.adInteger, whose value is 3, and maps onto ODBC’s DBTYPE_I4.

The concepts are largely the same, but the values are different. You have to do that mapping. IKVM can’t do it, because the .NET concepts don’t exist in the Java code it is translating.

> On Oct 18, 2025, at 7:16 PM, Jerome Haltom <jh...@alethic.solutions> wrote:
> 
> IKVM does 100%. But it is not a code translator. Byte code to IL.
> 
> We pass the full JDK test suite, for JDK8 anyways.
> 
> So you can kind of forget I'm using it, and pretend I'm just asking about Java, but I've got a custom driver framework that isn't JDBC, so I'm trying to do what Calcites JDBC driver does without JDBC.
> 
> The question is where in Calcite does the JDBC end and the Calcite begin, if you get my drift.
> 
> Get Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Julian Hyde <jh...@gmail.com>
> Sent: Saturday, October 18, 2025 9:04:56 PM
> To: dev@calcite.apache.org <de...@calcite.apache.org>
> Subject: Re: Calcite and !JDBC
> 
> It’s no accident that the comment at the top of CalcitePrepareImpl is “Shit just got real”. It’s the coordinator of the planning process, so needs to deal with both expression algebra and dynamics (generating a statement id, so that the statement can hook into resource and lifecycle management). It’s hard to cut this Gordian knot.
> 
> Can you mock the JDBC bits, or map them onto .NET concepts? (They’re not too dissimilar… both evolved from ODBC.)
> 
> If I were taking on this project, and using an automated translation tool like IKVM, I would make a clone of the Calcite Java code and refactor it a bit. I don’t know IKVM but I suspect that it can do 90% or 99% but struggles with the last 1%. Once you’ve discovered which refactorings made it work, you’ve discovered which are the right cuts in the Gordian knot, and can perhaps contribute those back to Calcite.
> 
> Julian
> 
> 
>> On Oct 17, 2025, at 9:09 AM, Jerome Haltom <jh...@alethic.solutions> wrote:
>> 
>> Hello.
>> 
>> I am doing an interesting project (or so I think) using Calcite. I am attempting to write a .NET library that makes use of Calcite. The goal being to expose the Calcite operations one can normally do under JDBC in a nice .NET-like API surface using System.Data. So, a .NET class, Apache.Calcite.Data.CalciteConnection, which implements the .NET driver/connection model, but ultimately does the same thing the Calcite JDBC driver does. A nice clean .NET-like API surface that uses Calcite under the covers. I am doing this with IKVM (which is a JVM "implementation" that converts Java bytecode to .NET IL).
>> 
>> I have no problem fully using all the Calcite classes this way. Including the JDBC driver, etc. Everything works perfectly fine. So, my question is not about this translation layer, or this JVM implementation, or anything like that. I only mention it to provide context to my question.
>> 
>> My question is about how 'deep' into the core Calcite code the JDBC stuff extends.
>> 
>> I started this effort out by figuring I would mostly just need to rewrite a more appropriate version of org.apache.calcite.jdbc, that ultimately surfaced a .NET API but used all the same Calcite internals. But the more I dig into that, the more I find a few JDBC-isms that have snuck outside that package. For instance, org.apache.calcite.jdbc.CalciteScema is a big class. Doing quite a bit of work. And it's referenced by org.apache.calcite.prepare. And it's also accessed by the SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some stuff in calcite.schema.impl.ViewTable, materialize, etc.
>> 
>> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm no longer just confronted with writing a new Connection-like implementation, but with new implementations of a number of other classes that seem pretty unrelated to JDBC itself.
>> 
>> So, then I think, okay. I may abandon that plan and instead preserve and not rewrite a class like CalciteSchema. But use the existing jdbc.CalciteSchema in my new CalciteConnection implementation. The namespaces/packages won't be as clear to users: they'll be creating jdbc.CalciteSchemas and adding them to Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should work.
>> 
>> So, how much of jdbc.* can I preserve, and how much is actually tied to JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not! CalcitePrepare has a few things in it, like CalciteSignature, which extends an Avatica class, Meta.Signature, and expects a list of AvaticaParameters. And this is very much JDBC-tied. But I need CalcitePrepare because of materialize/Lattice and all that.
>> 
>> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I can't make use it since it relies on having a bunch of Avatica JDBC specific stuff exposed. Which a non-JDBC access method wouldn't have.
>> 
>> Any recommendations here? Ignore my .NET related context if you want, and pretend I'm writing a driver for some other driver framework that just has no support for JDBC-like stuff.
>> 
>> 
> 


Re: Calcite and !JDBC

Posted by Jerome Haltom <jh...@alethic.solutions>.
IKVM does 100%. But it is not a code translator. Byte code to IL.

We pass the full JDK test suite, for JDK8 anyways.

So you can kind of forget I'm using it, and pretend I'm just asking about Java, but I've got a custom driver framework that isn't JDBC, so I'm trying to do what Calcites JDBC driver does without JDBC.

The question is where in Calcite does the JDBC end and the Calcite begin, if you get my drift.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Julian Hyde <jh...@gmail.com>
Sent: Saturday, October 18, 2025 9:04:56 PM
To: dev@calcite.apache.org <de...@calcite.apache.org>
Subject: Re: Calcite and !JDBC

It’s no accident that the comment at the top of CalcitePrepareImpl is “Shit just got real”. It’s the coordinator of the planning process, so needs to deal with both expression algebra and dynamics (generating a statement id, so that the statement can hook into resource and lifecycle management). It’s hard to cut this Gordian knot.

Can you mock the JDBC bits, or map them onto .NET concepts? (They’re not too dissimilar… both evolved from ODBC.)

If I were taking on this project, and using an automated translation tool like IKVM, I would make a clone of the Calcite Java code and refactor it a bit. I don’t know IKVM but I suspect that it can do 90% or 99% but struggles with the last 1%. Once you’ve discovered which refactorings made it work, you’ve discovered which are the right cuts in the Gordian knot, and can perhaps contribute those back to Calcite.

Julian


> On Oct 17, 2025, at 9:09 AM, Jerome Haltom <jh...@alethic.solutions> wrote:
>
> Hello.
>
> I am doing an interesting project (or so I think) using Calcite. I am attempting to write a .NET library that makes use of Calcite. The goal being to expose the Calcite operations one can normally do under JDBC in a nice .NET-like API surface using System.Data. So, a .NET class, Apache.Calcite.Data.CalciteConnection, which implements the .NET driver/connection model, but ultimately does the same thing the Calcite JDBC driver does. A nice clean .NET-like API surface that uses Calcite under the covers. I am doing this with IKVM (which is a JVM "implementation" that converts Java bytecode to .NET IL).
>
> I have no problem fully using all the Calcite classes this way. Including the JDBC driver, etc. Everything works perfectly fine. So, my question is not about this translation layer, or this JVM implementation, or anything like that. I only mention it to provide context to my question.
>
> My question is about how 'deep' into the core Calcite code the JDBC stuff extends.
>
> I started this effort out by figuring I would mostly just need to rewrite a more appropriate version of org.apache.calcite.jdbc, that ultimately surfaced a .NET API but used all the same Calcite internals. But the more I dig into that, the more I find a few JDBC-isms that have snuck outside that package. For instance, org.apache.calcite.jdbc.CalciteScema is a big class. Doing quite a bit of work. And it's referenced by org.apache.calcite.prepare. And it's also accessed by the SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some stuff in calcite.schema.impl.ViewTable, materialize, etc.
>
> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm no longer just confronted with writing a new Connection-like implementation, but with new implementations of a number of other classes that seem pretty unrelated to JDBC itself.
>
> So, then I think, okay. I may abandon that plan and instead preserve and not rewrite a class like CalciteSchema. But use the existing jdbc.CalciteSchema in my new CalciteConnection implementation. The namespaces/packages won't be as clear to users: they'll be creating jdbc.CalciteSchemas and adding them to Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should work.
>
> So, how much of jdbc.* can I preserve, and how much is actually tied to JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not! CalcitePrepare has a few things in it, like CalciteSignature, which extends an Avatica class, Meta.Signature, and expects a list of AvaticaParameters. And this is very much JDBC-tied. But I need CalcitePrepare because of materialize/Lattice and all that.
>
> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I can't make use it since it relies on having a bunch of Avatica JDBC specific stuff exposed. Which a non-JDBC access method wouldn't have.
>
> Any recommendations here? Ignore my .NET related context if you want, and pretend I'm writing a driver for some other driver framework that just has no support for JDBC-like stuff.
>
>


Re: Calcite and !JDBC

Posted by Julian Hyde <jh...@gmail.com>.
It’s no accident that the comment at the top of CalcitePrepareImpl is “Shit just got real”. It’s the coordinator of the planning process, so needs to deal with both expression algebra and dynamics (generating a statement id, so that the statement can hook into resource and lifecycle management). It’s hard to cut this Gordian knot.

Can you mock the JDBC bits, or map them onto .NET concepts? (They’re not too dissimilar… both evolved from ODBC.)

If I were taking on this project, and using an automated translation tool like IKVM, I would make a clone of the Calcite Java code and refactor it a bit. I don’t know IKVM but I suspect that it can do 90% or 99% but struggles with the last 1%. Once you’ve discovered which refactorings made it work, you’ve discovered which are the right cuts in the Gordian knot, and can perhaps contribute those back to Calcite. 

Julian


> On Oct 17, 2025, at 9:09 AM, Jerome Haltom <jh...@alethic.solutions> wrote:
> 
> Hello.
> 
> I am doing an interesting project (or so I think) using Calcite. I am attempting to write a .NET library that makes use of Calcite. The goal being to expose the Calcite operations one can normally do under JDBC in a nice .NET-like API surface using System.Data. So, a .NET class, Apache.Calcite.Data.CalciteConnection, which implements the .NET driver/connection model, but ultimately does the same thing the Calcite JDBC driver does. A nice clean .NET-like API surface that uses Calcite under the covers. I am doing this with IKVM (which is a JVM "implementation" that converts Java bytecode to .NET IL).
> 
> I have no problem fully using all the Calcite classes this way. Including the JDBC driver, etc. Everything works perfectly fine. So, my question is not about this translation layer, or this JVM implementation, or anything like that. I only mention it to provide context to my question.
> 
> My question is about how 'deep' into the core Calcite code the JDBC stuff extends.
> 
> I started this effort out by figuring I would mostly just need to rewrite a more appropriate version of org.apache.calcite.jdbc, that ultimately surfaced a .NET API but used all the same Calcite internals. But the more I dig into that, the more I find a few JDBC-isms that have snuck outside that package. For instance, org.apache.calcite.jdbc.CalciteScema is a big class. Doing quite a bit of work. And it's referenced by org.apache.calcite.prepare. And it's also accessed by the SqlSpatialTypeOperatorTable. And a few things in sql.validate. And some stuff in calcite.schema.impl.ViewTable, materialize, etc.
> 
> Basically, JDBC extends deeper into Calcite than I had expected. So, I'm no longer just confronted with writing a new Connection-like implementation, but with new implementations of a number of other classes that seem pretty unrelated to JDBC itself.
> 
> So, then I think, okay. I may abandon that plan and instead preserve and not rewrite a class like CalciteSchema. But use the existing jdbc.CalciteSchema in my new CalciteConnection implementation. The namespaces/packages won't be as clear to users: they'll be creating jdbc.CalciteSchemas and adding them to Apache.Cacite.Data.CalciteConnections. Bit funny looking. But should work.
> 
> So, how much of jdbc.* can I preserve, and how much is actually tied to JDBC? CalciteSchema seems pretty clear. But, CalcitePrepare does not! CalcitePrepare has a few things in it, like CalciteSignature, which extends an Avatica class, Meta.Signature, and expects a list of AvaticaParameters. And this is very much JDBC-tied. But I need CalcitePrepare because of materialize/Lattice and all that.
> 
> So, it looks like bunch of Core stuff is tied into CalcitePrepare. But I can't make use it since it relies on having a bunch of Avatica JDBC specific stuff exposed. Which a non-JDBC access method wouldn't have.
> 
> Any recommendations here? Ignore my .NET related context if you want, and pretend I'm writing a driver for some other driver framework that just has no support for JDBC-like stuff.
> 
>