You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "nehaljwani (via GitHub)" <gi...@apache.org> on 2023/06/04 10:52:20 UTC

[GitHub] [orc] nehaljwani opened a new pull request, #1529: ORC-1440: Check for protobuf config based module

nehaljwani opened a new pull request, #1529:
URL: https://github.com/apache/orc/pull/1529

   The recent versions of `libprotobuf` (>=4.22.0) provide their own CMake configs. We should check for the existence of the config and use the hand-rolled version as a fallback as it does not capture all deps.
   
   See also: https://gitlab.kitware.com/cmake/cmake/-/issues/24321


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1575601052

   Also, cc @wgtmac 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] wgtmac commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1579716417

   Could you please have a try with `-DBUILD_CPP_TESTS=ON -DBUILD_TOOLS=ON`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] nehaljwani commented on a diff in pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "nehaljwani (via GitHub)" <gi...@apache.org>.
nehaljwani commented on code in PR #1529:
URL: https://github.com/apache/orc/pull/1529#discussion_r1220528506


##########
cmake_modules/FindProtobuf.cmake:
##########
@@ -26,6 +26,24 @@
 # PROTOBUF_EXECUTABLE: location of protoc
 
 
+find_package (Protobuf CONFIG)
+if (Protobuf_FOUND)
+    set (PROTOBUF_LIBRARY protobuf::libprotobuf)
+    set (PROTOBUF_STATIC_LIB PROTOBUF_STATIC_LIB-NOTFOUND)
+    set (PROTOC_LIBRARY protobuf::libprotoc)
+    set (PROTOC_STATIC_LIB PROTOC_STATIC_LIB-NOTFOUND)
+    get_target_property (target_type protobuf::libprotobuf TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set(PROTOBUF_STATIC_LIB protobuf::libprotobuf)
+    endif ()
+    get_target_property (target_type protobuf::libprotoc TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set (PROTOC_STATIC_LIB protobuf::libprotoc)
+    endif ()
+    get_target_property (PROTOBUF_INCLUDE_DIR protobuf::libprotoc INTERFACE_INCLUDE_DIRECTORIES)
+    return ()

Review Comment:
   Updated in the revised commit. Now it prints:
   ```
   -- PROTOBUF_HOME: /path/to/libprotobuf
   -- Found the Protobuf headers: /path/to/libprotobuf/include
   -- Found the Protobuf library: protobuf::libprotobuf
   -- Found the Protoc library: protobuf::libprotoc
   -- Found the Protoc executable: protobuf::protoc
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on a diff in pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #1529:
URL: https://github.com/apache/orc/pull/1529#discussion_r1216779558


##########
cmake_modules/FindProtobuf.cmake:
##########
@@ -26,6 +26,24 @@
 # PROTOBUF_EXECUTABLE: location of protoc
 
 
+find_package(Protobuf CONFIG)
+if(Protobuf_FOUND)

Review Comment:
   Thank you for making a PR. Could you follow the existing coding style?
   
   https://github.com/apache/orc/blob/ec2ea9c6aff8b8515452df651f08695639c18cbb/cmake_modules/FindProtobuf.cmake#L74



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1585490215

   Thank you, @nehaljwani , @wgtmac , @stiga-huang . Merged to main for Apache ORC 2.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] stiga-huang commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "stiga-huang (via GitHub)" <gi...@apache.org>.
stiga-huang commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1576753174

   > I also maintain the conda-forge recipe for orc. The latest build there was successful with this patch and protobuf v4.23.2
   
   Do you build the entire orc lib (including tools) or just e.g. liborc.a? Just curious if all the functionality work with protobuf v4.23.2. If so, we can add tests to guard them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] nehaljwani commented on a diff in pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "nehaljwani (via GitHub)" <gi...@apache.org>.
nehaljwani commented on code in PR #1529:
URL: https://github.com/apache/orc/pull/1529#discussion_r1220527922


##########
cmake_modules/FindProtobuf.cmake:
##########
@@ -26,6 +26,24 @@
 # PROTOBUF_EXECUTABLE: location of protoc
 
 
+find_package (Protobuf CONFIG)
+if (Protobuf_FOUND)
+    set (PROTOBUF_LIBRARY protobuf::libprotobuf)
+    set (PROTOBUF_STATIC_LIB PROTOBUF_STATIC_LIB-NOTFOUND)
+    set (PROTOC_LIBRARY protobuf::libprotoc)
+    set (PROTOC_STATIC_LIB PROTOC_STATIC_LIB-NOTFOUND)
+    get_target_property (target_type protobuf::libprotobuf TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set(PROTOBUF_STATIC_LIB protobuf::libprotobuf)
+    endif ()
+    get_target_property (target_type protobuf::libprotoc TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set (PROTOC_STATIC_LIB protobuf::libprotoc)
+    endif ()
+    get_target_property (PROTOBUF_INCLUDE_DIR protobuf::libprotoc INTERFACE_INCLUDE_DIRECTORIES)

Review Comment:
   Updated in the revised commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] nehaljwani commented on a diff in pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "nehaljwani (via GitHub)" <gi...@apache.org>.
nehaljwani commented on code in PR #1529:
URL: https://github.com/apache/orc/pull/1529#discussion_r1216785128


##########
cmake_modules/FindProtobuf.cmake:
##########
@@ -26,6 +26,24 @@
 # PROTOBUF_EXECUTABLE: location of protoc
 
 
+find_package(Protobuf CONFIG)
+if(Protobuf_FOUND)

Review Comment:
   I've updated the commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] wgtmac commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1576966832

   BTW, I was thinking the possibility to migrate the proto file to use proto3 in v2.0.0 once we split the repo for format. @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] dongjoon-hyun closed pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #1529: ORC-1440: Check for protobuf config based module
URL: https://github.com/apache/orc/pull/1529


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] nehaljwani commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "nehaljwani (via GitHub)" <gi...@apache.org>.
nehaljwani commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1575608734

   I also maintain the conda-forge recipe for orc. The latest build there was successful with this patch and protobuf v4.23.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] stiga-huang commented on a diff in pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "stiga-huang (via GitHub)" <gi...@apache.org>.
stiga-huang commented on code in PR #1529:
URL: https://github.com/apache/orc/pull/1529#discussion_r1218002691


##########
cmake_modules/FindProtobuf.cmake:
##########
@@ -26,6 +26,24 @@
 # PROTOBUF_EXECUTABLE: location of protoc
 
 
+find_package (Protobuf CONFIG)
+if (Protobuf_FOUND)
+    set (PROTOBUF_LIBRARY protobuf::libprotobuf)
+    set (PROTOBUF_STATIC_LIB PROTOBUF_STATIC_LIB-NOTFOUND)
+    set (PROTOC_LIBRARY protobuf::libprotoc)
+    set (PROTOC_STATIC_LIB PROTOC_STATIC_LIB-NOTFOUND)
+    get_target_property (target_type protobuf::libprotobuf TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set(PROTOBUF_STATIC_LIB protobuf::libprotobuf)
+    endif ()
+    get_target_property (target_type protobuf::libprotoc TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set (PROTOC_STATIC_LIB protobuf::libprotoc)
+    endif ()
+    get_target_property (PROTOBUF_INCLUDE_DIR protobuf::libprotoc INTERFACE_INCLUDE_DIRECTORIES)

Review Comment:
    `PROTOBUF_EXECUTABLE` is missing here. Is it set in protobuf's own CMake configs?



##########
cmake_modules/FindProtobuf.cmake:
##########
@@ -26,6 +26,24 @@
 # PROTOBUF_EXECUTABLE: location of protoc
 
 
+find_package (Protobuf CONFIG)
+if (Protobuf_FOUND)
+    set (PROTOBUF_LIBRARY protobuf::libprotobuf)
+    set (PROTOBUF_STATIC_LIB PROTOBUF_STATIC_LIB-NOTFOUND)
+    set (PROTOC_LIBRARY protobuf::libprotoc)
+    set (PROTOC_STATIC_LIB PROTOC_STATIC_LIB-NOTFOUND)
+    get_target_property (target_type protobuf::libprotobuf TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set(PROTOBUF_STATIC_LIB protobuf::libprotobuf)
+    endif ()
+    get_target_property (target_type protobuf::libprotoc TYPE)
+    if (target_type STREQUAL "STATIC_LIBRARY")
+        set (PROTOC_STATIC_LIB protobuf::libprotoc)
+    endif ()
+    get_target_property (PROTOBUF_INCLUDE_DIR protobuf::libprotoc INTERFACE_INCLUDE_DIRECTORIES)
+    return ()

Review Comment:
   Before `return`, it'd be nice to print the variables as we did at line 93 - 102. Or will the protobuf's config print some info?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] nehaljwani commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "nehaljwani (via GitHub)" <gi...@apache.org>.
nehaljwani commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1579591053

   >Do you build the entire orc lib (including tools) or just e.g. liborc.a? Just curious if all the functionality work with protobuf v4.23.2. If so, we can add tests to guard them.
   
   I used the following steps locally (not sure if it covers all targets):
   ```
   cmake -S/path/to/orc -B/tmp/build -GNinja \
      -DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
      -DPROTOBUF_HOME=$CONDA_PREFIX \
      -DBUILD_JAVA=OFF \
      -DZLIB_HOME=$CONDA_PREFIX \
      -DSNAPPY_HOME=$CONDA_PREFIX \
      -DLZ4_HOME=$CONDA_PREFIX \
      -DZSTD_HOME=$CONDA_PREFIX \
      -DSTOP_BUILD_ON_WARNING=OFF \
      -DBUILD_CPP_TESTS=OFF
   
   cmake --build /tmp/build
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] nehaljwani commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "nehaljwani (via GitHub)" <gi...@apache.org>.
nehaljwani commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1581598968

   Done, the build succeeds with both options turned on with my patch applied.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] wgtmac commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1576962995

   Probably it is a good time to upgrade protobuf version as well. @stiga-huang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [orc] wgtmac commented on pull request #1529: ORC-1440: Check for protobuf config based module

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1529:
URL: https://github.com/apache/orc/pull/1529#issuecomment-1575607327

   Thanks @nehaljwani and @dongjoon-hyun!
   
   These changes look fine to me. However, we are still on a very old version of protobuf (3.5.1). I am not sure if it will work properly with higher versions of protobuf. I have seen issues when Apache Arrow tried to compile ORC with a different protobuf version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org