You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Paul Nienaber <Pa...@improving.com.INVALID> on 2023/02/13 19:05:02 UTC
Build issues (Protobuf internal symbols)
Hi folks,
I'm not familiar yet with all of the details of how Protobuf is set up to be built and included or whether there's a subtle difference between the CMake targets (headers vs cc sources), but we've recently gone and started touching some of the Protobuf types from a C++ header and are seeing build issues stemming from Protobuf-internal symbols/macros not being properly in scope.
Working with /cpp/src/arrow/flight/sql/client.h we've included "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but using <google/protobuf/any.pb.g was not exposing generated types from sql/FlightSql), and we're seeing the following:
[1/4] Building CXX object src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
FAILED: src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
/opt/homebrew/bin/ccache /Library/Developer/CommandLineTools/usr/bin/c++ -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS -DARROW_WITH_ZLIB -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental -DGRPC_USE_CERTIFICATE_VERIFIER -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src -I/Users/pauln/bq/git/arrow/cpp/src -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem /opt/homebrew/include -isystem /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem /opt/homebrew/opt/openssl@1.1/include -isystem /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces -Wno-unused-parameter -Wno-constant-logical-operand -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC -std=c++17 -MD -MT src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o -MF src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d -o src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
In file included from /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
In file included from /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3: error: expected expression
static const uint32_t offsets[];
^
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32: error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
^
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8: note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
^
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73: error: no member named 'DescriptorTable' in namespace 'google::protobuf::internal'
ARROW_FLIGHT_SQL_EXPORT extern const ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable descriptor_table_FlightSql_2eproto;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88: error: expected ';' after top level declarator
ARROW_FLIGHT_SQL_EXPORT extern const ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable descriptor_table_FlightSql_2eproto;
[...]
Any insight or suggestions are welcome here!
Cheers
Paul
Re: Build issues (Protobuf internal symbols)
Posted by Yaron Gvili <rt...@hotmail.com>.
@Li, my understanding is that if you generate headers from the same Arrow protobuf files using the same toolchain (including protoc) used in the Arrow build, then you will be able to use these headers to correctly access protobuf objects within Arrow structures. This doesn't unhide symbols in the Arrow libraries but just recreates them in your binaries. In other words, you'd get ABI compatibility.
Yaron.
________________________________
From: David Li <li...@apache.org>
Sent: Monday, February 13, 2023 11:59 PM
To: dev@arrow.apache.org <de...@arrow.apache.org>
Subject: Re: Build issues (Protobuf internal symbols)
"Yes", except Arrow has a linker script that hides the symbols, so you would have to carve out an exception to get it to work. If you unhide one symbol, you can make it work again (though I don't think this will let you actually use the Protobuf types, only write your own Protobufs referencing Arrow Protobuf types): https://github.com/lidavidm/arrow-14771-demo
Also on Windows, it just cannot work, because protoc doesn't properly implement support for injecting a dllimport/dllexport visibility macro (even though it claims to), at least that was the case last I tried.
On Mon, Feb 13, 2023, at 16:49, Li Jin wrote:
> "
> In this case though, it's just that we purposely hide symbols by default.
> If there's a use case, we could unhide this specific symbol (we did it for
> one other Protobuf symbol) which would let you externally generate and use
> the headers (as long as you take care not to actually include the generated
> sources).
> "
> My knowledge about these symbols hiding is lacky so I would like to take
> this opportunity and learn about it if possible.
>
> How is the hide/unhide done here? My understanding is that these protobuf
> symbols are included in the libarrow_fight.so, but not exposed via any
> public header, and therefore it is "hidden". But if the user protoc the
> header files, they can still access the symbols in libarrow_fight.so?
>
>
> On Mon, Feb 13, 2023 at 3:28 PM Paul Nienaber
> <Pa...@improving.com.invalid> wrote:
>
>> Yes I can build normally. Good point about this being in a public header,
>> as it would consequently be in any further application translation units...
>> I'll look into whether we can refactor what's been done already.
>> ________________________________
>> From: David Li <li...@apache.org>
>> Sent: Monday, February 13, 2023 12:15 PM
>> To: dev@arrow.apache.org <de...@arrow.apache.org>
>> Subject: Re: Build issues (Protobuf internal symbols)
>>
>> Oh, sorry, this is for Arrow development...
>>
>> A public header should not include an internal header (this includes the
>> Protobuf-generated headers). But can you build the project without
>> modification already?
>>
>> On Mon, Feb 13, 2023, at 15:02, Paul Nienaber wrote:
>> > Hi David,
>> >
>> > This is for Arrow feature development, where the changes have been made
>> > in client.h. Am I possibly running into symbol visibility guards I'm
>> > not expecting here and didn't spot yet?
>> >
>> > Paul
>> > ________________________________
>> > From: David Li <li...@apache.org>
>> > Sent: Monday, February 13, 2023 11:11 AM
>> > To: dev@arrow.apache.org <de...@arrow.apache.org>
>> > Subject: Re: Build issues (Protobuf internal symbols)
>> >
>> > The Protobuf-generated code from Arrow is not meant for external
>> > consumption, and in general, the way Protobuf works will make this
>> > complicated. What is the use case here?
>> >
>> > (In fact, that header shouldn't even be available for you to include.
>> > It seems you're linking/including it from a build of Arrow, but it
>> > won't be there once Arrow is actually installed.)
>> >
>> > In this case though, it's just that we purposely hide symbols by
>> > default. If there's a use case, we could unhide this specific symbol
>> > (we did it for one other Protobuf symbol) which would let you
>> > externally generate and use the headers (as long as you take care not
>> > to actually include the generated sources).
>> >
>> > On Mon, Feb 13, 2023, at 14:05, Paul Nienaber wrote:
>> >> Hi folks,
>> >>
>> >> I'm not familiar yet with all of the details of how Protobuf is set up
>> >> to be built and included or whether there's a subtle difference between
>> >> the CMake targets (headers vs cc sources), but we've recently gone and
>> >> started touching some of the Protobuf types from a C++ header and are
>> >> seeing build issues stemming from Protobuf-internal symbols/macros not
>> >> being properly in scope.
>> >>
>> >> Working with /cpp/src/arrow/flight/sql/client.h we've included
>> >> "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but
>> >> using <google/protobuf/any.pb.g was not exposing generated types from
>> >> sql/FlightSql), and we're seeing the following:
>> >>
>> >> [1/4] Building CXX object
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> FAILED:
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> /opt/homebrew/bin/ccache
>> >> /Library/Developer/CommandLineTools/usr/bin/c++
>> >> -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING
>> >> -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS
>> >> -DARROW_WITH_ZLIB
>> >> -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental
>> >> -DGRPC_USE_CERTIFICATE_VERIFIER
>> >> -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD
>> >> -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src
>> >> -I/Users/pauln/bq/git/arrow/cpp/src
>> >> -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem
>> >> /opt/homebrew/include -isystem
>> >> /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem
>> >> /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem
>> >> /opt/homebrew/opt/openssl@1.1/include -isystem
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
>> >> -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra
>> >> -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces
>> >> -Wno-unused-parameter -Wno-constant-logical-operand
>> >> -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed
>> >> -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot
>> >> /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC
>> >> -std=c++17 -MD -MT
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> -MF
>> >>
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d
>> >> -o
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
>> >> In file included from
>> >> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
>> >> In file included from
>> >> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
>> >> error: expected expression
>> >> static const uint32_t offsets[];
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
>> >> error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
>> >> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
>> >> note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
>> >> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
>> >> error: no member named 'DescriptorTable' in namespace
>> >> 'google::protobuf::internal'
>> >> ARROW_FLIGHT_SQL_EXPORT extern const
>> >> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> >> descriptor_table_FlightSql_2eproto;
>> >>
>> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
>> >> error: expected ';' after top level declarator
>> >> ARROW_FLIGHT_SQL_EXPORT extern const
>> >> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> >> descriptor_table_FlightSql_2eproto;
>> >> [...]
>> >>
>> >>
>> >> Any insight or suggestions are welcome here!
>> >>
>> >> Cheers
>> >> Paul
>>
Re: Build issues (Protobuf internal symbols)
Posted by David Li <li...@apache.org>.
"Yes", except Arrow has a linker script that hides the symbols, so you would have to carve out an exception to get it to work. If you unhide one symbol, you can make it work again (though I don't think this will let you actually use the Protobuf types, only write your own Protobufs referencing Arrow Protobuf types): https://github.com/lidavidm/arrow-14771-demo
Also on Windows, it just cannot work, because protoc doesn't properly implement support for injecting a dllimport/dllexport visibility macro (even though it claims to), at least that was the case last I tried.
On Mon, Feb 13, 2023, at 16:49, Li Jin wrote:
> "
> In this case though, it's just that we purposely hide symbols by default.
> If there's a use case, we could unhide this specific symbol (we did it for
> one other Protobuf symbol) which would let you externally generate and use
> the headers (as long as you take care not to actually include the generated
> sources).
> "
> My knowledge about these symbols hiding is lacky so I would like to take
> this opportunity and learn about it if possible.
>
> How is the hide/unhide done here? My understanding is that these protobuf
> symbols are included in the libarrow_fight.so, but not exposed via any
> public header, and therefore it is "hidden". But if the user protoc the
> header files, they can still access the symbols in libarrow_fight.so?
>
>
> On Mon, Feb 13, 2023 at 3:28 PM Paul Nienaber
> <Pa...@improving.com.invalid> wrote:
>
>> Yes I can build normally. Good point about this being in a public header,
>> as it would consequently be in any further application translation units...
>> I'll look into whether we can refactor what's been done already.
>> ________________________________
>> From: David Li <li...@apache.org>
>> Sent: Monday, February 13, 2023 12:15 PM
>> To: dev@arrow.apache.org <de...@arrow.apache.org>
>> Subject: Re: Build issues (Protobuf internal symbols)
>>
>> Oh, sorry, this is for Arrow development...
>>
>> A public header should not include an internal header (this includes the
>> Protobuf-generated headers). But can you build the project without
>> modification already?
>>
>> On Mon, Feb 13, 2023, at 15:02, Paul Nienaber wrote:
>> > Hi David,
>> >
>> > This is for Arrow feature development, where the changes have been made
>> > in client.h. Am I possibly running into symbol visibility guards I'm
>> > not expecting here and didn't spot yet?
>> >
>> > Paul
>> > ________________________________
>> > From: David Li <li...@apache.org>
>> > Sent: Monday, February 13, 2023 11:11 AM
>> > To: dev@arrow.apache.org <de...@arrow.apache.org>
>> > Subject: Re: Build issues (Protobuf internal symbols)
>> >
>> > The Protobuf-generated code from Arrow is not meant for external
>> > consumption, and in general, the way Protobuf works will make this
>> > complicated. What is the use case here?
>> >
>> > (In fact, that header shouldn't even be available for you to include.
>> > It seems you're linking/including it from a build of Arrow, but it
>> > won't be there once Arrow is actually installed.)
>> >
>> > In this case though, it's just that we purposely hide symbols by
>> > default. If there's a use case, we could unhide this specific symbol
>> > (we did it for one other Protobuf symbol) which would let you
>> > externally generate and use the headers (as long as you take care not
>> > to actually include the generated sources).
>> >
>> > On Mon, Feb 13, 2023, at 14:05, Paul Nienaber wrote:
>> >> Hi folks,
>> >>
>> >> I'm not familiar yet with all of the details of how Protobuf is set up
>> >> to be built and included or whether there's a subtle difference between
>> >> the CMake targets (headers vs cc sources), but we've recently gone and
>> >> started touching some of the Protobuf types from a C++ header and are
>> >> seeing build issues stemming from Protobuf-internal symbols/macros not
>> >> being properly in scope.
>> >>
>> >> Working with /cpp/src/arrow/flight/sql/client.h we've included
>> >> "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but
>> >> using <google/protobuf/any.pb.g was not exposing generated types from
>> >> sql/FlightSql), and we're seeing the following:
>> >>
>> >> [1/4] Building CXX object
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> FAILED:
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> /opt/homebrew/bin/ccache
>> >> /Library/Developer/CommandLineTools/usr/bin/c++
>> >> -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING
>> >> -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS
>> >> -DARROW_WITH_ZLIB
>> >> -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental
>> >> -DGRPC_USE_CERTIFICATE_VERIFIER
>> >> -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD
>> >> -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src
>> >> -I/Users/pauln/bq/git/arrow/cpp/src
>> >> -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem
>> >> /opt/homebrew/include -isystem
>> >> /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem
>> >> /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem
>> >> /opt/homebrew/opt/openssl@1.1/include -isystem
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
>> >> -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra
>> >> -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces
>> >> -Wno-unused-parameter -Wno-constant-logical-operand
>> >> -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed
>> >> -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot
>> >> /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC
>> >> -std=c++17 -MD -MT
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> -MF
>> >>
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d
>> >> -o
>> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> >> -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
>> >> In file included from
>> >> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
>> >> In file included from
>> >> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
>> >> error: expected expression
>> >> static const uint32_t offsets[];
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
>> >> error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
>> >> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
>> >> note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
>> >> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> >> ^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
>> >> error: no member named 'DescriptorTable' in namespace
>> >> 'google::protobuf::internal'
>> >> ARROW_FLIGHT_SQL_EXPORT extern const
>> >> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> >> descriptor_table_FlightSql_2eproto;
>> >>
>> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
>> >>
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
>> >> error: expected ';' after top level declarator
>> >> ARROW_FLIGHT_SQL_EXPORT extern const
>> >> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> >> descriptor_table_FlightSql_2eproto;
>> >> [...]
>> >>
>> >>
>> >> Any insight or suggestions are welcome here!
>> >>
>> >> Cheers
>> >> Paul
>>
Re: Build issues (Protobuf internal symbols)
Posted by Li Jin <ic...@gmail.com>.
"
In this case though, it's just that we purposely hide symbols by default.
If there's a use case, we could unhide this specific symbol (we did it for
one other Protobuf symbol) which would let you externally generate and use
the headers (as long as you take care not to actually include the generated
sources).
"
My knowledge about these symbols hiding is lacky so I would like to take
this opportunity and learn about it if possible.
How is the hide/unhide done here? My understanding is that these protobuf
symbols are included in the libarrow_fight.so, but not exposed via any
public header, and therefore it is "hidden". But if the user protoc the
header files, they can still access the symbols in libarrow_fight.so?
On Mon, Feb 13, 2023 at 3:28 PM Paul Nienaber
<Pa...@improving.com.invalid> wrote:
> Yes I can build normally. Good point about this being in a public header,
> as it would consequently be in any further application translation units...
> I'll look into whether we can refactor what's been done already.
> ________________________________
> From: David Li <li...@apache.org>
> Sent: Monday, February 13, 2023 12:15 PM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Subject: Re: Build issues (Protobuf internal symbols)
>
> Oh, sorry, this is for Arrow development...
>
> A public header should not include an internal header (this includes the
> Protobuf-generated headers). But can you build the project without
> modification already?
>
> On Mon, Feb 13, 2023, at 15:02, Paul Nienaber wrote:
> > Hi David,
> >
> > This is for Arrow feature development, where the changes have been made
> > in client.h. Am I possibly running into symbol visibility guards I'm
> > not expecting here and didn't spot yet?
> >
> > Paul
> > ________________________________
> > From: David Li <li...@apache.org>
> > Sent: Monday, February 13, 2023 11:11 AM
> > To: dev@arrow.apache.org <de...@arrow.apache.org>
> > Subject: Re: Build issues (Protobuf internal symbols)
> >
> > The Protobuf-generated code from Arrow is not meant for external
> > consumption, and in general, the way Protobuf works will make this
> > complicated. What is the use case here?
> >
> > (In fact, that header shouldn't even be available for you to include.
> > It seems you're linking/including it from a build of Arrow, but it
> > won't be there once Arrow is actually installed.)
> >
> > In this case though, it's just that we purposely hide symbols by
> > default. If there's a use case, we could unhide this specific symbol
> > (we did it for one other Protobuf symbol) which would let you
> > externally generate and use the headers (as long as you take care not
> > to actually include the generated sources).
> >
> > On Mon, Feb 13, 2023, at 14:05, Paul Nienaber wrote:
> >> Hi folks,
> >>
> >> I'm not familiar yet with all of the details of how Protobuf is set up
> >> to be built and included or whether there's a subtle difference between
> >> the CMake targets (headers vs cc sources), but we've recently gone and
> >> started touching some of the Protobuf types from a C++ header and are
> >> seeing build issues stemming from Protobuf-internal symbols/macros not
> >> being properly in scope.
> >>
> >> Working with /cpp/src/arrow/flight/sql/client.h we've included
> >> "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but
> >> using <google/protobuf/any.pb.g was not exposing generated types from
> >> sql/FlightSql), and we're seeing the following:
> >>
> >> [1/4] Building CXX object
> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> >> FAILED:
> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> >> /opt/homebrew/bin/ccache
> >> /Library/Developer/CommandLineTools/usr/bin/c++
> >> -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING
> >> -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS
> >> -DARROW_WITH_ZLIB
> >> -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental
> >> -DGRPC_USE_CERTIFICATE_VERIFIER
> >> -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD
> >> -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src
> >> -I/Users/pauln/bq/git/arrow/cpp/src
> >> -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem
> >> /opt/homebrew/include -isystem
> >> /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem
> >> /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem
> >> /opt/homebrew/opt/openssl@1.1/include -isystem
> >>
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
> >> -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra
> >> -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces
> >> -Wno-unused-parameter -Wno-constant-logical-operand
> >> -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed
> >> -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot
> >> /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC
> >> -std=c++17 -MD -MT
> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> >> -MF
> >>
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d
> >> -o
> >> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> >> -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
> >> In file included from
> >> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
> >> In file included from
> >> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
> >>
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
> >> error: expected expression
> >> static const uint32_t offsets[];
> >> ^
> >>
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
> >> error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
> >> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
> >> ^
> >>
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
> >> note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
> >> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
> >> ^
> >>
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
> >> error: no member named 'DescriptorTable' in namespace
> >> 'google::protobuf::internal'
> >> ARROW_FLIGHT_SQL_EXPORT extern const
> >> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
> >> descriptor_table_FlightSql_2eproto;
> >>
> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
> >>
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
> >> error: expected ';' after top level declarator
> >> ARROW_FLIGHT_SQL_EXPORT extern const
> >> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
> >> descriptor_table_FlightSql_2eproto;
> >> [...]
> >>
> >>
> >> Any insight or suggestions are welcome here!
> >>
> >> Cheers
> >> Paul
>
Re: Build issues (Protobuf internal symbols)
Posted by Paul Nienaber <Pa...@improving.com.INVALID>.
Yes I can build normally. Good point about this being in a public header, as it would consequently be in any further application translation units... I'll look into whether we can refactor what's been done already.
________________________________
From: David Li <li...@apache.org>
Sent: Monday, February 13, 2023 12:15 PM
To: dev@arrow.apache.org <de...@arrow.apache.org>
Subject: Re: Build issues (Protobuf internal symbols)
Oh, sorry, this is for Arrow development...
A public header should not include an internal header (this includes the Protobuf-generated headers). But can you build the project without modification already?
On Mon, Feb 13, 2023, at 15:02, Paul Nienaber wrote:
> Hi David,
>
> This is for Arrow feature development, where the changes have been made
> in client.h. Am I possibly running into symbol visibility guards I'm
> not expecting here and didn't spot yet?
>
> Paul
> ________________________________
> From: David Li <li...@apache.org>
> Sent: Monday, February 13, 2023 11:11 AM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Subject: Re: Build issues (Protobuf internal symbols)
>
> The Protobuf-generated code from Arrow is not meant for external
> consumption, and in general, the way Protobuf works will make this
> complicated. What is the use case here?
>
> (In fact, that header shouldn't even be available for you to include.
> It seems you're linking/including it from a build of Arrow, but it
> won't be there once Arrow is actually installed.)
>
> In this case though, it's just that we purposely hide symbols by
> default. If there's a use case, we could unhide this specific symbol
> (we did it for one other Protobuf symbol) which would let you
> externally generate and use the headers (as long as you take care not
> to actually include the generated sources).
>
> On Mon, Feb 13, 2023, at 14:05, Paul Nienaber wrote:
>> Hi folks,
>>
>> I'm not familiar yet with all of the details of how Protobuf is set up
>> to be built and included or whether there's a subtle difference between
>> the CMake targets (headers vs cc sources), but we've recently gone and
>> started touching some of the Protobuf types from a C++ header and are
>> seeing build issues stemming from Protobuf-internal symbols/macros not
>> being properly in scope.
>>
>> Working with /cpp/src/arrow/flight/sql/client.h we've included
>> "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but
>> using <google/protobuf/any.pb.g was not exposing generated types from
>> sql/FlightSql), and we're seeing the following:
>>
>> [1/4] Building CXX object
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> FAILED:
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> /opt/homebrew/bin/ccache
>> /Library/Developer/CommandLineTools/usr/bin/c++
>> -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING
>> -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS
>> -DARROW_WITH_ZLIB
>> -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental
>> -DGRPC_USE_CERTIFICATE_VERIFIER
>> -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD
>> -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src
>> -I/Users/pauln/bq/git/arrow/cpp/src
>> -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem
>> /opt/homebrew/include -isystem
>> /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem
>> /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem
>> /opt/homebrew/opt/openssl@1.1/include -isystem
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
>> -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra
>> -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces
>> -Wno-unused-parameter -Wno-constant-logical-operand
>> -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed
>> -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot
>> /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC
>> -std=c++17 -MD -MT
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> -MF
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d
>> -o
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
>> In file included from
>> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
>> In file included from
>> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
>> error: expected expression
>> static const uint32_t offsets[];
>> ^
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
>> error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
>> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> ^
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
>> note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
>> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> ^
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
>> error: no member named 'DescriptorTable' in namespace
>> 'google::protobuf::internal'
>> ARROW_FLIGHT_SQL_EXPORT extern const
>> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> descriptor_table_FlightSql_2eproto;
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
>> error: expected ';' after top level declarator
>> ARROW_FLIGHT_SQL_EXPORT extern const
>> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> descriptor_table_FlightSql_2eproto;
>> [...]
>>
>>
>> Any insight or suggestions are welcome here!
>>
>> Cheers
>> Paul
Re: Build issues (Protobuf internal symbols)
Posted by David Li <li...@apache.org>.
Oh, sorry, this is for Arrow development...
A public header should not include an internal header (this includes the Protobuf-generated headers). But can you build the project without modification already?
On Mon, Feb 13, 2023, at 15:02, Paul Nienaber wrote:
> Hi David,
>
> This is for Arrow feature development, where the changes have been made
> in client.h. Am I possibly running into symbol visibility guards I'm
> not expecting here and didn't spot yet?
>
> Paul
> ________________________________
> From: David Li <li...@apache.org>
> Sent: Monday, February 13, 2023 11:11 AM
> To: dev@arrow.apache.org <de...@arrow.apache.org>
> Subject: Re: Build issues (Protobuf internal symbols)
>
> The Protobuf-generated code from Arrow is not meant for external
> consumption, and in general, the way Protobuf works will make this
> complicated. What is the use case here?
>
> (In fact, that header shouldn't even be available for you to include.
> It seems you're linking/including it from a build of Arrow, but it
> won't be there once Arrow is actually installed.)
>
> In this case though, it's just that we purposely hide symbols by
> default. If there's a use case, we could unhide this specific symbol
> (we did it for one other Protobuf symbol) which would let you
> externally generate and use the headers (as long as you take care not
> to actually include the generated sources).
>
> On Mon, Feb 13, 2023, at 14:05, Paul Nienaber wrote:
>> Hi folks,
>>
>> I'm not familiar yet with all of the details of how Protobuf is set up
>> to be built and included or whether there's a subtle difference between
>> the CMake targets (headers vs cc sources), but we've recently gone and
>> started touching some of the Protobuf types from a C++ header and are
>> seeing build issues stemming from Protobuf-internal symbols/macros not
>> being properly in scope.
>>
>> Working with /cpp/src/arrow/flight/sql/client.h we've included
>> "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but
>> using <google/protobuf/any.pb.g was not exposing generated types from
>> sql/FlightSql), and we're seeing the following:
>>
>> [1/4] Building CXX object
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> FAILED:
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> /opt/homebrew/bin/ccache
>> /Library/Developer/CommandLineTools/usr/bin/c++
>> -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING
>> -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS
>> -DARROW_WITH_ZLIB
>> -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental
>> -DGRPC_USE_CERTIFICATE_VERIFIER
>> -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD
>> -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src
>> -I/Users/pauln/bq/git/arrow/cpp/src
>> -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem
>> /opt/homebrew/include -isystem
>> /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem
>> /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem
>> /opt/homebrew/opt/openssl@1.1/include -isystem
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
>> -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra
>> -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces
>> -Wno-unused-parameter -Wno-constant-logical-operand
>> -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed
>> -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot
>> /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC
>> -std=c++17 -MD -MT
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> -MF
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d
>> -o
>> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
>> -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
>> In file included from
>> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
>> In file included from
>> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
>> error: expected expression
>> static const uint32_t offsets[];
>> ^
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
>> error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
>> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> ^
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
>> note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
>> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
>> ^
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
>> error: no member named 'DescriptorTable' in namespace
>> 'google::protobuf::internal'
>> ARROW_FLIGHT_SQL_EXPORT extern const
>> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> descriptor_table_FlightSql_2eproto;
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
>> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
>> error: expected ';' after top level declarator
>> ARROW_FLIGHT_SQL_EXPORT extern const
>> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
>> descriptor_table_FlightSql_2eproto;
>> [...]
>>
>>
>> Any insight or suggestions are welcome here!
>>
>> Cheers
>> Paul
Re: Build issues (Protobuf internal symbols)
Posted by Paul Nienaber <Pa...@improving.com.INVALID>.
Hi David,
This is for Arrow feature development, where the changes have been made in client.h. Am I possibly running into symbol visibility guards I'm not expecting here and didn't spot yet?
Paul
________________________________
From: David Li <li...@apache.org>
Sent: Monday, February 13, 2023 11:11 AM
To: dev@arrow.apache.org <de...@arrow.apache.org>
Subject: Re: Build issues (Protobuf internal symbols)
The Protobuf-generated code from Arrow is not meant for external consumption, and in general, the way Protobuf works will make this complicated. What is the use case here?
(In fact, that header shouldn't even be available for you to include. It seems you're linking/including it from a build of Arrow, but it won't be there once Arrow is actually installed.)
In this case though, it's just that we purposely hide symbols by default. If there's a use case, we could unhide this specific symbol (we did it for one other Protobuf symbol) which would let you externally generate and use the headers (as long as you take care not to actually include the generated sources).
On Mon, Feb 13, 2023, at 14:05, Paul Nienaber wrote:
> Hi folks,
>
> I'm not familiar yet with all of the details of how Protobuf is set up
> to be built and included or whether there's a subtle difference between
> the CMake targets (headers vs cc sources), but we've recently gone and
> started touching some of the Protobuf types from a C++ header and are
> seeing build issues stemming from Protobuf-internal symbols/macros not
> being properly in scope.
>
> Working with /cpp/src/arrow/flight/sql/client.h we've included
> "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but
> using <google/protobuf/any.pb.g was not exposing generated types from
> sql/FlightSql), and we're seeing the following:
>
> [1/4] Building CXX object
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> FAILED:
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> /opt/homebrew/bin/ccache
> /Library/Developer/CommandLineTools/usr/bin/c++
> -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING
> -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS
> -DARROW_WITH_ZLIB
> -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental
> -DGRPC_USE_CERTIFICATE_VERIFIER
> -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD
> -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src
> -I/Users/pauln/bq/git/arrow/cpp/src
> -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem
> /opt/homebrew/include -isystem
> /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem
> /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem
> /opt/homebrew/opt/openssl@1.1/include -isystem
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
> -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra
> -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces
> -Wno-unused-parameter -Wno-constant-logical-operand
> -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed
> -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot
> /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC
> -std=c++17 -MD -MT
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> -MF
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d
> -o
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
> In file included from
> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
> In file included from
> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
> error: expected expression
> static const uint32_t offsets[];
> ^
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
> error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
> ^
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
> note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
> ^
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
> error: no member named 'DescriptorTable' in namespace
> 'google::protobuf::internal'
> ARROW_FLIGHT_SQL_EXPORT extern const
> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
> descriptor_table_FlightSql_2eproto;
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
> error: expected ';' after top level declarator
> ARROW_FLIGHT_SQL_EXPORT extern const
> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
> descriptor_table_FlightSql_2eproto;
> [...]
>
>
> Any insight or suggestions are welcome here!
>
> Cheers
> Paul
Re: Build issues (Protobuf internal symbols)
Posted by David Li <li...@apache.org>.
The Protobuf-generated code from Arrow is not meant for external consumption, and in general, the way Protobuf works will make this complicated. What is the use case here?
(In fact, that header shouldn't even be available for you to include. It seems you're linking/including it from a build of Arrow, but it won't be there once Arrow is actually installed.)
In this case though, it's just that we purposely hide symbols by default. If there's a use case, we could unhide this specific symbol (we did it for one other Protobuf symbol) which would let you externally generate and use the headers (as long as you take care not to actually include the generated sources).
On Mon, Feb 13, 2023, at 14:05, Paul Nienaber wrote:
> Hi folks,
>
> I'm not familiar yet with all of the details of how Protobuf is set up
> to be built and included or whether there's a subtle difference between
> the CMake targets (headers vs cc sources), but we've recently gone and
> started touching some of the Protobuf types from a C++ header and are
> seeing build issues stemming from Protobuf-internal symbols/macros not
> being properly in scope.
>
> Working with /cpp/src/arrow/flight/sql/client.h we've included
> "arrow/flight/sql/FlightSql.pb.h" (this may itself be an issue, but
> using <google/protobuf/any.pb.g was not exposing generated types from
> sql/FlightSql), and we're seeing the following:
>
> [1/4] Building CXX object
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> FAILED:
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> /opt/homebrew/bin/ccache
> /Library/Developer/CommandLineTools/usr/bin/c++
> -DARROW_EXTRA_ERROR_CONTEXT -DARROW_FLIGHT_SQL_EXPORTING
> -DARROW_HAVE_NEON -DARROW_WITH_RE2 -DARROW_WITH_TIMING_TESTS
> -DARROW_WITH_ZLIB
> -DGRPC_NAMESPACE_FOR_TLS_CREDENTIALS_OPTIONS=grpc::experimental
> -DGRPC_USE_CERTIFICATE_VERIFIER
> -DGRPC_USE_TLS_CHANNEL_CREDENTIALS_OPTIONS -DURI_STATIC_BUILD
> -I/Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src
> -I/Users/pauln/bq/git/arrow/cpp/src
> -I/Users/pauln/bq/git/arrow/cpp/src/generated -isystem
> /opt/homebrew/include -isystem
> /Users/pauln/bq/git/arrow/cpp/thirdparty/flatbuffers/include -isystem
> /Users/pauln/bq/git/arrow/cpp/thirdparty/hadoop/include -isystem
> /opt/homebrew/opt/openssl@1.1/include -isystem
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/jemalloc_ep-prefix/src
> -fno-aligned-new -Qunused-arguments -fcolor-diagnostics -Wall -Wextra
> -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces
> -Wno-unused-parameter -Wno-constant-logical-operand
> -Wno-return-stack-address -Wno-unknown-warning-option -Wno-pass-failed
> -march=armv8-a -g -Werror -O0 -ggdb -arch arm64 -isysroot
> /Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk -fPIC
> -std=c++17 -MD -MT
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> -MF
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o.d
> -o
> src/arrow/flight/sql/CMakeFiles/arrow_flight_sql_objlib.dir/client.cc.o
> -c /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc
> In file included from
> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.cc:22:
> In file included from
> /Users/pauln/bq/git/arrow/cpp/src/arrow/flight/sql/client.h:24:
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:47:3:
> error: expected expression
> static const uint32_t offsets[];
> ^
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:32:
> error: variable has incomplete type 'struct ARROW_FLIGHT_SQL_EXPORT'
> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
> ^
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:46:8:
> note: forward declaration of 'ARROW_FLIGHT_SQL_EXPORT'
> struct ARROW_FLIGHT_SQL_EXPORT TableStruct_FlightSql_2eproto {
> ^
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:73:
> error: no member named 'DescriptorTable' in namespace
> 'google::protobuf::internal'
> ARROW_FLIGHT_SQL_EXPORT extern const
> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
> descriptor_table_FlightSql_2eproto;
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
> /Users/pauln/bq/git/arrow/cpp/session-managemen-build-debug/src/arrow/flight/sql/FlightSql.pb.h:49:88:
> error: expected ';' after top level declarator
> ARROW_FLIGHT_SQL_EXPORT extern const
> ::PROTOBUF_NAMESPACE_ID::internal::DescriptorTable
> descriptor_table_FlightSql_2eproto;
> [...]
>
>
> Any insight or suggestions are welcome here!
>
> Cheers
> Paul