You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/12 14:40:01 UTC

[GitHub] [arrow] lidavidm opened a new issue, #33633: [C++] Separate Protobuf dependencies into a `libarrow_proto.so`

lidavidm opened a new issue, #33633:
URL: https://github.com/apache/arrow/issues/33633

   ### Describe the enhancement requested
   
   In #14082 we found that conflicts between different Arrow libraries using Protobuf would lead to runtime crashes. This mostly affects macOS wheels, where:
   
   1. Arrow is built as a shared library,
   2. Arrow dependencies are linked statically,
   3. The macOS linker doesn't support version scripts.
   
   In this case, `libarrow.so`, `libarrow_flight.so`, `libarrow_flight_sql.so`, and `libarrow_substrait.so` all link their own copies of `libprotobuf.a`. (`libarrow.so` due to ORC.) And they also all export those symbols. This happens to have been mostly fine so far since none of them use the same Protobuf definitions, **except** Flight SQL and Substrait both use `google.protobuf.Any`. So when both are loaded into a process, Protobuf triggers an assertion because `Any` is defined twice.
   
   The solutions would be:
   - Rewrite Protobuf definitions to not use Any. However this is a source-breaking change and is probably not going to happen on the Substrait side.
   - Configure the macOS linker to hide the Protobuf symbols. I've seen it suggested that this is OK because then there will be multiple copies of the Protobuf runtime and they shouldn't conflict. That's why Linux and Windows wheels seemed to be unaffected. However, the macOS linker doesn't support version scripts, and it's very hard to convince CMake to use `-hidden-l` (the equivalent flag) - and there's no native support for it in CMake so you end up having to rewrite/refactor a lot of CMake to distinguish between targets and link commands (which is not usual). (I _think_ you could do this by adding a custom target, adding the 'real' protobuf as a non-link dependency, and adding the linker flag to the custom target's properties? But you'd still have to be careful in places where the Arrow CMake scripts list a bunch of targets to bundle or export - those still need to point to the 'real' Protobuf target somehow.)
   - Create a new shared library that links Protobuf and contains all the Protobuf definitions. Then all other libraries depend on this instead of depending on this directly. My worry is Windows, where protoc doesn't support generating dllexport/dllimport annotations, so you need something like [`WINDOWS_EXPORT_ALL_SYMBOLS`](https://cmake.org/cmake/help/latest/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html#prop_tgt:WINDOWS_EXPORT_ALL_SYMBOLS). 
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on issue #33633: [C++] Separate Protobuf dependencies into a `libarrow_proto.so`

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on issue #33633:
URL: https://github.com/apache/arrow/issues/33633#issuecomment-1385690840

   On the PR, you mentioned at some point (idea to link libprotobuf dynamically with the several libarrows instead of statically as we do for all non-arrow dependencies):
   
   > Ah, digging into the build - the issue is that they both link libprotobuf statically. Instead, we need to link it dynamically. The awkward part will be getting vcpkg to do this (I don't think we can make vcpkg build just one library dynamically? And then we also have to remember to bundle it into the wheel).
   
   I don't know if this option is still being considered, but just to note about vcpk: this should be possible by defining a custom triplet, where you can change the linking for specific packages (https://vcpkg.io/en/docs/users/triplets.html#per-port-customization). And we already have custom triplet files, which currently configures all libraries to use static linking. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] lidavidm commented on issue #33633: [C++] Separate Protobuf dependencies into a `libarrow_proto.so`

Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #33633:
URL: https://github.com/apache/arrow/issues/33633#issuecomment-1381161354

   Ah, thanks for reminding me. That's an option, I believe, but what I could find seemed waffly about whether it would actually solve this. (Also: the conflict is in a core Protobuf type, so I'm not sure if 'lite' applies to this? I guess you do link a different library, so maybe...)
   
   I'm not sure if there is a practical way to get CMake to do this - it seems like CMake has no notion of "different linker flags besides `-l`" which is what I actually want. Short of that, it's all just one big hack.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on issue #33633: [C++] Separate Protobuf dependencies into a `libarrow_proto.so`

Posted by GitBox <gi...@apache.org>.
westonpace commented on issue #33633:
URL: https://github.com/apache/arrow/issues/33633#issuecomment-1381101118

   Configuring the linker to hide the symbols would be my preference I think.  Pity this is all so complicated.  Did we give up on the "protobuf lite" option?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] lidavidm commented on issue #33633: [C++] Separate Protobuf dependencies into a `libarrow_proto.so`

Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #33633:
URL: https://github.com/apache/arrow/issues/33633#issuecomment-1381161497

   That said, I can investigate more - just not in time for the release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] lidavidm commented on issue #33633: [C++] Separate Protobuf dependencies into a `libarrow_proto.so`

Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #33633:
URL: https://github.com/apache/arrow/issues/33633#issuecomment-1385909842

   No longer a worry since we're going to punt on having the C++ Flight SQL implementation available to Python. I'll leave this issue open, though, since the conflict may come up again in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Separate Protobuf dependencies into a `libarrow_proto.so` [arrow]

Posted by "yilei (via GitHub)" <gi...@apache.org>.
yilei commented on issue #33633:
URL: https://github.com/apache/arrow/issues/33633#issuecomment-2935812259

   Looks like this would solve a different issue mentioned here: https://github.com/apache/arrow/issues/40088
   
   Are you aware of workarounds for folks who run into this on macOS?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Separate Protobuf dependencies into a `libarrow_proto.so` [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33633:
URL: https://github.com/apache/arrow/issues/33633#issuecomment-4622013884

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Separate Protobuf dependencies into a `libarrow_proto.so` [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33633: [C++] Separate Protobuf dependencies into a `libarrow_proto.so`
URL: https://github.com/apache/arrow/issues/33633


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org