You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by "metegenez (via GitHub)" <gi...@apache.org> on 2026/03/30 10:55:03 UTC
[D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
GitHub user metegenez created a discussion: CGO dependency as adoption blocker for C++ engines
Hey folks,
I've been working on integrating ADBC into a StarRocks. The ADBC C API and driver manager are great, clean, embeddable, easy to work with.
The problem is that the moment you need an actual driver (Flight SQL, Snowflake, etc.), you hit a CGO dependency. For C++ query engines, this is a real friction point:
- Extra Go toolchain in an already complex C++ build system
- Go runtime (GC, goroutine scheduler) embedded inside a latency-sensitive C++ process
- Cross-compilation and static linking headaches
- -
I know there was an earlier C++ Flight SQL driver attempt (apache/arrow#14082) that was dropped in favor of the Go implementation. Since then the c/driver/framework has matured a lot, and Arrow C++ Flight libraries are solid.
Would the community be open to pure C++ driver implementations, starting with Flight SQL? I'm an Arrow committer and happy to help the work, just want to check if this direction makes sense before diving in.
I never worked on this library before, so there are a lot of experienced people here to guide me at least.
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "metegenez (via GitHub)" <gi...@apache.org>.
GitHub user metegenez edited a discussion: CGO dependency as adoption blocker for C++ engines
Hey folks,
I've been working on integrating ADBC into a StarRocks. The ADBC C API and driver manager are great, clean, embeddable, easy to work with.
The problem is that the moment you need an actual driver (Flight SQL, Snowflake, etc.), you hit a CGO dependency. For C++ query engines, this is a real friction point:
- Extra Go toolchain in an already complex C++ build system
- Go runtime (GC, goroutine scheduler) embedded inside a latency-sensitive C++ process
- Cross-compilation and static linking headaches
I know there was an earlier C++ Flight SQL driver attempt (apache/arrow#14082) that was dropped in favor of the Go implementation. Since then the c/driver/framework has matured a lot, and Arrow C++ Flight libraries are solid.
Would the community be open to pure C++ driver implementations, starting with Flight SQL? I'm an Arrow committer and happy to help the work, just want to check if this direction makes sense before diving in.
I never worked on this library before, so there are a lot of experienced people here to guide me at least.
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
GitHub user lidavidm edited a comment on the discussion: CGO dependency as adoption blocker for C++ engines
The C++ implementation hit unsolvable dependency issues, unfortunately. If you ever plan on using Protobuf, gRPC, or possibly arrow-cpp inside your application, it's possible you'll run into the same issues.
Could you expand on the latency concern? I'm not sure how the Go runtime would add meaningful latency compared to, say, sending a query across the network to the cloud, waiting for it, and fetching the result. (If you're worried about the Go runtime taking resources from the host, we can explore ways to control that / gRPC-C++ itself effectively has a runtime with its own thread pools, too; and I'm not sure a driver built with either would be actually unloadable/dlclose-able due to global state.)
That said, if there's enough contributors willing to maintain a C++ version of the Flight SQL driver, I don't think I'd say there's no possibility; I'd be curious if you plan on porting _all_ the drivers, though, and what you envision for build/test/distribution.
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156#discussioncomment-16379590
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
GitHub user zeroshade added a comment to the discussion: CGO dependency as adoption blocker for C++ engines
> * Extra Go toolchain in an already complex C++ build system
> * Cross-compilation and static linking headaches
Another question I have is that the point of using ADBC, like ODBC before it, is so that you *don't* have to statically link the driver in. Instead you would use the `AdbcDriverManager` and then just load the driver on the fly so that you can work with any driver instead of being tied to only a single one. Is there a reason you're trying to include the Go Snowflake ADBC driver in your build process rather than just treating the .so as a black box to dynamically load by pulling it from a release? (either by `dbc` or by a Github release, etc.)
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156#discussioncomment-16383411
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
GitHub user zeroshade added a comment to the discussion: CGO dependency as adoption blocker for C++ engines
For the purposes of integrating ADBC, the fact that a particular driver might be using cgo under the hood is honestly irrelevant. Either the driver is performant enough or not, but that has no bearing on the overall utility of ADBC as a whole (plenty of drivers available, only a subset of them use Go/cgo).
The entire point of using the pre-built .so files and driver manager at runtime is that it doesn't matter what the driver is actually written using.
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156#discussioncomment-16456520
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "metegenez (via GitHub)" <gi...@apache.org>.
GitHub user metegenez edited a comment on the discussion: CGO dependency as adoption blocker for C++ engines
Thanks for the context on the protobuf/gRPC issues, that makes sense, I'll avoid the C++ driver path than. https://github.com/StarRocks/starrocks/pull/70297 I tried to push ADBC into Starrocks, but i feel there is a pushback. What do you think on this? Maybe I should improve how I am loading the library? I guess I did a bit wrong on static linking here? https://github.com/StarRocks/starrocks/pull/70297/changes#diff-803104e5c983ecaeacc02d30e00b0406b63c3179f268bc67228e348e7cf9779bR103 , I couldnt make dlopen work.
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156#discussioncomment-16391828
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
GitHub user lidavidm added a comment to the discussion: CGO dependency as adoption blocker for C++ engines
The C++ implementation hit unsolvable dependency issues, unfortunately. If you ever plan on using Protobuf, gRPC, or possibly arrow-cpp inside your application, it's possible you'll run into the same issues.
Could you expand on the latency concern? I'm not sure how the Go runtime would add meaningful latency compared to, say, sending a query across the network to the cloud, waiting for it, and fetching the result. (If you're worried about the Go runtime taking resources from the host, we can explore ways to control that / gRPC-C++ itself effectively has a runtime with its own thread pools, too; and I'm not sure a driver built with either would be actually unloadable/dlclose-able due to global state.)
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156#discussioncomment-16379590
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "metegenez (via GitHub)" <gi...@apache.org>.
GitHub user metegenez added a comment to the discussion: CGO dependency as adoption blocker for C++ engines
Did I feel the pushback unnecessarily, or is there a real reason behind it? I'd to use ADBC on starrocks but I am having hard time to address the concern.
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156#discussioncomment-16391953
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "metegenez (via GitHub)" <gi...@apache.org>.
GitHub user metegenez added a comment to the discussion: CGO dependency as adoption blocker for C++ engines
Thanks for the context on the protobuf/gRPC issues, that makes sense, I'll avoid the C++ driver path than. https://github.com/StarRocks/starrocks/pull/70297 I tried to push ADBC into Starrocks, but i feel there is a pushback. What do you think on this? Maybe I should improve how I am loading the library? I guess I did a bit wrong on static linking here? https://github.com/StarRocks/starrocks/pull/70297/changes#diff-803104e5c983ecaeacc02d30e00b0406b63c3179f268bc67228e348e7cf9779bR107-R127 , I couldnt make dlopen work.
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156#discussioncomment-16391828
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] CGO dependency as adoption blocker for C++ engines [arrow-adbc]
Posted by "metegenez (via GitHub)" <gi...@apache.org>.
GitHub user metegenez added a comment to the discussion: CGO dependency as adoption blocker for C++ engines
You're right and that's exactly the approach I'm planning to take. Build-time is solved with pre-built .so + driver manager, no argument there. How about runtime? What trade-offs should i know by using cgo underhood?
GitHub link: https://github.com/apache/arrow-adbc/discussions/4156#discussioncomment-16391872
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org