You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/06 18:20:57 UTC
[GitHub] [arrow] lidavidm opened a new pull request #10260: ARROW-12671: [C++][Python][FlightRPC] WIP: Simple OpenTelemetry integration
lidavidm opened a new pull request #10260:
URL: https://github.com/apache/arrow/pull/10260
As discussed on the mailing list, this is a proof-of-concept that integrates OpenTelemetry for (distributed) tracing. This supports the following:
* OTel is used as a header-only library by default.
* Optionally, Arrow will link to OTel and configure it based on an environment variable.
* OTel is used to instrument parts of the async scanner, and Flight.
* For Flight, clients will export spans in outgoing headers; servers will inject spans from incoming headers, and start new spans on each call.
* OTel is checked for in Python. If present, the necessary middleware is added.
* The Python Flight server propagates spans from C++ to Python. Nothing propagates spans in the other direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-970588263
@github-actions crossbow submit almalinux-8-amd64 centos-8-amd64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-970588038
```
No such option: -g
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-984007670
I'm curious: why are gRPC and protobuf required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r739891424
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
Any chance you can do the same thing that's done elsewhere in this file and call `find_package` if `OpenTelemetry_SOURCE == "SYSTEM"`?
Bundling unconditionally like this makes it really difficult to deal OpenTelemetry provided by a package manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r753535272
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3865,6 +3902,202 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+
+ build_nlohmann_json_once()
+ find_curl()
+
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api ext sdk)
+ set(_OPENTELEMETRY_LIBS
+ common
+ http_client_curl
+ ostream_span_exporter
+ otlp_http_exporter
+ otlp_recordable
+ proto
+ resources
+ trace
+ version)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ if(_OPENTELEMETRY_LIB STREQUAL "http_client_curl")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "ostream_span_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_ostream_span${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "otlp_http_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_otlp_http${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ else()
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ endif()
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST)
+ # Don't specify the DEPENDS unless we actually have dependencies, else
+ # Ninja/other build systems may consider this target to always be dirty
+ set(_OPENTELEMETRY_DEPENDENCIES)
+ add_custom_target(opentelemetry_dependencies)
+
+ set(_OPENTELEMETRY_DEPENDENCIES "opentelemetry_dependencies")
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ list(APPEND OPENTELEMETRY_PREFIX_PATH_LIST ${NLOHMANN_JSON_PREFIX})
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_OTLP=ON
Review comment:
Could you use `list(APPEND ...)` for all `OPENTELEMETRY_CMAKE_ARGS` changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-983989733
Benchmark runs are scheduled for baseline = 200f1679f9837a83b105ba081f31ca47cbe22de1 and contender = bca06811e2da06a8c22de9ac0921329b29829521. bca06811e2da06a8c22de9ac0921329b29829521 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a8779635087d4dca91f81e1b36c31064...89f0b858dcbc41518455b77fb925f92a/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/261584bbf92946cdad09914abd83954a...1d80c3c857cd4839809462f04cef946b/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/84b1c177605b4549b54e8e8c4a09ed54...a841851bef404cdea0a686613f2f35c3/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-983989733
Benchmark runs are scheduled for baseline = 200f1679f9837a83b105ba081f31ca47cbe22de1 and contender = bca06811e2da06a8c22de9ac0921329b29829521. bca06811e2da06a8c22de9ac0921329b29829521 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a8779635087d4dca91f81e1b36c31064...89f0b858dcbc41518455b77fb925f92a/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/261584bbf92946cdad09914abd83954a...1d80c3c857cd4839809462f04cef946b/)
[Finished :arrow_down:0.44% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/84b1c177605b4549b54e8e8c4a09ed54...a841851bef404cdea0a686613f2f35c3/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-970668332
@github-actions crossbow submit -g nightly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r749724689
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
Oh, actually, we did talk about it [a little bit](https://lists.apache.org/thread/2wj87fw2vmokx4pmhm1whr6gt79wmqyb), but that was back in June and it was not obvious that the use case was different. However, I'll still poke the ML, and I'll improve things here so we can build without OpenTelemetry entirely (which would sidestep the RPM issues too, though I'd like to figure that out still) since it'll be more helpful for development (for now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++][Python][FlightRPC] WIP: Simple OpenTelemetry integration
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-833754832
ML link for discussion: https://lists.apache.org/thread.html/re0db1b4d7a82f4cdb7bfab4bdd3620a9b3813a22626800d9772ccbda%40%3Cdev.arrow.apache.org%3E
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r736935110
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
And being able to use the API only would simplify our build (we could even perhaps vendor the headers?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-967350943
Revision: 3e242a23d2238df7f846587e9a29768e8602659e
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1128](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1128)
|Task|Status|
|----|------|
|almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-almalinux-8-amd64)|
|almalinux-8-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|amazon-linux-2-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-amazon-linux-2-amd64)|
|centos-7-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-centos-7-amd64)|
|centos-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-centos-8-amd64)|
|centos-8-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|conda-clean|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-clean)|
|conda-linux-gcc-py310-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py310-arm64)|
|conda-linux-gcc-py310-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py310-cuda)|
|conda-linux-gcc-py36-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py36-arm64)|
|conda-linux-gcc-py36-cpu-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py36-cpu-r40)|
|conda-linux-gcc-py36-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py36-cuda)|
|conda-linux-gcc-py37-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py37-arm64)|
|conda-linux-gcc-py37-cpu-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py37-cpu-r41)|
|conda-linux-gcc-py37-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py37-cuda)|
|conda-linux-gcc-py38-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py38-arm64)|
|conda-linux-gcc-py38-cpu|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py38-cpu)|
|conda-linux-gcc-py38-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py38-cuda)|
|conda-linux-gcc-py39-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py39-arm64)|
|conda-linux-gcc-py39-cpu|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py39-cpu)|
|conda-linux-gcc-py39-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-linux-gcc-py39-cuda)|
|conda-osx-arm64-clang-py38|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-osx-arm64-clang-py38)|
|conda-osx-arm64-clang-py39|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-osx-arm64-clang-py39)|
|conda-osx-clang-py36-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-osx-clang-py36-r40)|
|conda-osx-clang-py37-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-osx-clang-py37-r41)|
|conda-osx-clang-py38|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-osx-clang-py38)|
|conda-osx-clang-py39|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-osx-clang-py39)|
|conda-win-vs2017-py36-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-win-vs2017-py36-r40)|
|conda-win-vs2017-py37-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-win-vs2017-py37-r41)|
|conda-win-vs2017-py38|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-win-vs2017-py38)|
|conda-win-vs2017-py39|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-conda-win-vs2017-py39)|
|debian-bookworm-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-debian-bookworm-amd64)|
|debian-bookworm-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|debian-bullseye-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-debian-bullseye-amd64)|
|debian-bullseye-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|debian-buster-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-debian-buster-amd64)|
|debian-buster-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-example-cpp-minimal-build-static)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-example-cpp-minimal-build-static-system-dependency)|
|homebrew-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-homebrew-cpp)|
|homebrew-r-autobrew|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-homebrew-r-autobrew)|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-java-jars)|
|nuget|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-nuget)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-python-sdist)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-build-cpp-fuzz)|
|test-build-vcpkg-win|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-build-vcpkg-win)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-conda-cpp-valgrind)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.10)|
|test-conda-python-3.6|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.6)|
|test-conda-python-3.6-pandas-0.23|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.6-pandas-0.23)|
|test-conda-python-3.7|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.7)|
|test-conda-python-3.7-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.7-hdfs-2.9.2)|
|test-conda-python-3.7-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.7-hdfs-3.2.1)|
|test-conda-python-3.7-kartothek-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.7-kartothek-latest)|
|test-conda-python-3.7-kartothek-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.7-kartothek-master)|
|test-conda-python-3.7-pandas-0.24|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.7-pandas-0.24)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.7-pandas-latest)|
|test-conda-python-3.7-spark-v3.1.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.7-spark-v3.1.2)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.8)|
|test-conda-python-3.8-hypothesis|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.8-hypothesis)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.8-pandas-latest)|
|test-conda-python-3.8-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.8-pandas-nightly)|
|test-conda-python-3.8-spark-v3.2.0|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.8-spark-v3.2.0)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.9)|
|test-conda-python-3.9-dask-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.9-dask-latest)|
|test-conda-python-3.9-dask-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.9-dask-master)|
|test-conda-python-3.9-pandas-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.9-pandas-master)|
|test-conda-python-3.9-spark-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-conda-python-3.9-spark-master)|
|test-debian-10-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-debian-10-cpp)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-debian-11-cpp)|
|test-debian-11-go-1.15|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-debian-11-go-1.15)|
|test-debian-11-python-3|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-debian-11-python-3)|
|test-debian-c-glib|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-debian-c-glib)|
|test-debian-ruby|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-debian-ruby)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-fedora-33-cpp)|
|test-fedora-33-python-3|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-fedora-33-python-3)|
|test-fedora-r-clang-sanitizer|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-fedora-r-clang-sanitizer)|
|test-r-arrow-backwards-compatibility|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-arrow-backwards-compatibility)|
|test-r-depsource-auto|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-depsource-auto)|
|test-r-depsource-system|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-depsource-system)|
|test-r-devdocs|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-devdocs)|
|test-r-gcc-11|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-gcc-11)|
|test-r-install-local|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-install-local)|
|test-r-linux-as-cran|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-linux-as-cran)|
|test-r-linux-rchk|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-linux-rchk)|
|test-r-linux-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-linux-valgrind)|
|test-r-minimal-build|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-minimal-build)|
|test-r-offline-maximal|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-offline-maximal)|
|test-r-offline-minimal|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-offline-minimal)|
|test-r-rhub-debian-gcc-devel-lto-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-rhub-debian-gcc-devel-lto-latest)|
|test-r-rhub-ubuntu-gcc-release-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-rhub-ubuntu-gcc-release-latest)|
|test-r-rocker-r-base-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-rocker-r-base-latest)|
|test-r-rstudio-r-base-3.6-bionic|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-rstudio-r-base-3.6-bionic)|
|test-r-rstudio-r-base-3.6-centos7-devtoolset-8|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-rstudio-r-base-3.6-centos7-devtoolset-8)|
|test-r-rstudio-r-base-3.6-centos8|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-rstudio-r-base-3.6-centos8)|
|test-r-rstudio-r-base-3.6-opensuse15|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-rstudio-r-base-3.6-opensuse15)|
|test-r-rstudio-r-base-3.6-opensuse42|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-r-rstudio-r-base-3.6-opensuse42)|
|test-r-ubuntu-21.04|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-ubuntu-21.04)|
|test-r-versions|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-r-versions)|
|test-skyhook-integration|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-skyhook-integration)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-18.04-python-3|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-ubuntu-18.04-python-3)|
|test-ubuntu-18.04-r-sanitizer|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-ubuntu-18.04-r-sanitizer)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-20.04-cpp-bundled)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
|test-ubuntu-20.10-docs|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-ubuntu-20.10-docs)|
|test-ubuntu-c-glib|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-c-glib)|
|test-ubuntu-default-docs|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1128-azure-test-ubuntu-default-docs)|
|test-ubuntu-ruby|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-test-ubuntu-ruby)|
|ubuntu-bionic-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-ubuntu-bionic-amd64)|
|ubuntu-bionic-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|ubuntu-focal-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-ubuntu-focal-amd64)|
|ubuntu-focal-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|ubuntu-hirsute-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-ubuntu-hirsute-amd64)|
|ubuntu-hirsute-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|ubuntu-impish-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-ubuntu-impish-amd64)|
|ubuntu-impish-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-macos-big-sur-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-big-sur-cp310-arm64)|
|wheel-macos-big-sur-cp310-universal2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-big-sur-cp310-universal2)|
|wheel-macos-big-sur-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-big-sur-cp38-arm64)|
|wheel-macos-big-sur-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-big-sur-cp39-arm64)|
|wheel-macos-big-sur-cp39-universal2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-big-sur-cp39-universal2)|
|wheel-macos-high-sierra-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-high-sierra-cp310-amd64)|
|wheel-macos-high-sierra-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-high-sierra-cp36-amd64)|
|wheel-macos-high-sierra-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-high-sierra-cp37-amd64)|
|wheel-macos-high-sierra-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-high-sierra-cp38-amd64)|
|wheel-macos-high-sierra-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-high-sierra-cp39-amd64)|
|wheel-macos-mavericks-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-mavericks-cp310-amd64)|
|wheel-macos-mavericks-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-mavericks-cp36-amd64)|
|wheel-macos-mavericks-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-mavericks-cp37-amd64)|
|wheel-macos-mavericks-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-mavericks-cp38-amd64)|
|wheel-macos-mavericks-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-macos-mavericks-cp39-amd64)|
|wheel-manylinux2010-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2010-cp310-amd64)|
|wheel-manylinux2010-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2010-cp36-amd64)|
|wheel-manylinux2010-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2010-cp37-amd64)|
|wheel-manylinux2010-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2010-cp38-amd64)|
|wheel-manylinux2010-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2010-cp39-amd64)|
|wheel-manylinux2014-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2014-cp310-amd64)|
|wheel-manylinux2014-cp310-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2014-cp36-amd64)|
|wheel-manylinux2014-cp36-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2014-cp37-amd64)|
|wheel-manylinux2014-cp37-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2014-cp38-amd64)|
|wheel-manylinux2014-cp38-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-manylinux2014-cp39-amd64)|
|wheel-manylinux2014-cp39-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-windows-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-windows-cp310-amd64)|
|wheel-windows-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-windows-cp36-amd64)|
|wheel-windows-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-windows-cp37-amd64)|
|wheel-windows-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-windows-cp38-amd64)|
|wheel-windows-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1128-github-wheel-windows-cp39-amd64)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r753515902
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3661,17 +3711,7 @@ macro(build_google_cloud_cpp_storage)
add_dependencies(google_cloud_cpp_dependencies nlohmann_json_ep)
# Typically the steps to build the AWKSSDK provide `CURL::libcurl`, but if that is
# disabled we need to provide our own.
- if(NOT TARGET CURL::libcurl)
- find_package(CURL REQUIRED)
- if(NOT TARGET CURL::libcurl)
- # For CMake 3.11 or older
- add_library(CURL::libcurl UNKNOWN IMPORTED)
- set_target_properties(CURL::libcurl
- PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
- "${CURL_INCLUDE_DIRS}" IMPORTED_LOCATION
- "${CURL_LIBRARIES}")
- endif()
- endif()
+ find_curl()
Review comment:
Can we remove this `find_curl()` and the above `Typically ...` comment for cURL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-969355064
> https://github.com/ursacomputing/crossbow/runs/4193651802?check_suite_focus=true#step:8:1827 shows `ninja install` runs steps for `opentelemetry_ep`. This may be related. Generally, `ninja install` doesn't need to run steps for `opentelemetry_ep`.
Hmm, indeed, thanks for spotting that. I'll take a further look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956320657
I've enabled the OTLP build, with caveats:
- It will not work with bundled gRPC/Protobuf since OTel appears not to declare a dependency on gRPC from its Protobuf module. (I need to replicate and report upstream, though I'm having issues getting OTel to build in general right now.)
- Even though we enable only the OTLP HTTP exporter here, we still need gRPC because the upstream build always depends on both.
- I haven't actually wired up the exporter for use yet.
I'll also get it building against "system" OTel next (assuming you specify it via CMAKE_PREFIX_PATH, since we don't have Conda/vcpkg/system package manager packages).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956330534
@lidavidm Do you have an example somewhere? I am happy to help try to spin things up and see where I get stuck (my success there wouldn't block this PR in any way)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r753523207
##########
File path: cpp/cmake_modules/DefineOptions.cmake
##########
@@ -371,6 +371,9 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
define_option(ARROW_WITH_BACKTRACE "Build with backtrace support" ON)
+ define_option(ARROW_WITH_OPENTELEMETRY
+ "Build libraries with OpenTelemetry support for distributed tracing" OFF)
Review comment:
Hmm, perhaps. It seems a little odd since it's not its own top-level directory (and not its own user-facing API) like the other components generally are, and I think we'd want ARROW_ENGINE and perhaps ARROW_FLIGHT to require OpenTelemetry at some point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r753521988
##########
File path: cpp/cmake_modules/DefineOptions.cmake
##########
@@ -371,6 +371,9 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
define_option(ARROW_WITH_BACKTRACE "Build with backtrace support" ON)
+ define_option(ARROW_WITH_OPENTELEMETRY
+ "Build libraries with OpenTelemetry support for distributed tracing" OFF)
Review comment:
How about adding `ARROW_TRACING` (or something) option to the "Project component" section instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956333167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-967331477
What is left to do on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-952268885
The build error on AppVeyor _should_ be fixed by https://github.com/apache/arrow/pull/11547.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r738744932
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
The issue that will come up is applications that mix Python libraries and arrow c++ code, that already have exporters and are running in some arbitrary configuration. I guess those folks can build arrow without the exporter and use that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm edited a comment on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm edited a comment on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956320657
I've enabled the OTLP build, with caveats:
- It will not work with bundled gRPC/Protobuf since OTel appears not to declare a dependency on gRPC from its Protobuf module. (I need to replicate and report upstream, though I'm having issues getting OTel to build in general right now.)
- Even though we enable only the OTLP HTTP exporter here, we still need gRPC because the upstream build always depends on both.
- I haven't actually wired up the exporter for use yet.
I'll also get it building against "system" OTel next (assuming you specify it via CMAKE_PREFIX_PATH, since we don't have Conda/vcpkg/system package manager packages).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r759664202
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -286,6 +294,9 @@ endif()
if(ARROW_FLIGHT)
set(ARROW_WITH_GRPC ON)
+endif()
+
+if(ARROW_WITH_GRPC)
# gRPC requires zlib
Review comment:
Can we remove this comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-984011595
It would be nice to make gRPC at least optional, as it's a rather heavy dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r749742583
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
I think at some point we may end up with a new "query engine" module which includes things like the substrait consumer and data holder (these components need to reference datasets and parquet and so can't be placed in the compute module). The "query engine" module would then include OpenTelemetry not just as a development module but in production too. I see the ability to generate statistics on running queries to be an essential part of a query engine and not just a development profiling feature.
Maybe then we could enable OT only if "flight" or "query engine" were enabled? Either way, I agree this is a good discussion for the ML.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r750707704
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3865,6 +3905,221 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api ext sdk)
+ set(_OPENTELEMETRY_LIBS
+ common
+ http_client_curl
+ ostream_span_exporter
+ otlp_http_exporter
+ otlp_recordable
+ proto
+ resources
+ trace
+ version)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ if(_OPENTELEMETRY_LIB STREQUAL "http_client_curl")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "ostream_span_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_ostream_span${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "otlp_http_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_otlp_http${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ else()
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ endif()
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST)
+ # Don't specify the DEPENDS unless we actually have dependencies, else
+ # Ninja/other build systems may consider this target to always be dirty
+ set(_OPENTELEMETRY_DEPENDENCIES)
+ add_custom_target(opentelemetry_dependencies)
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ set(_OPENTELEMETRY_DEPENDENCIES "opentelemetry_dependencies")
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ list(APPEND OPENTELEMETRY_PREFIX_PATH_LIST ${NLOHMANN_JSON_PREFIX})
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_OTLP=ON
+ -DWITH_OTLP_HTTP=ON -DWITH_OTLP_GRPC=OFF)
+
+ # OpenTelemetry with OTLP enabled requires Protobuf definitions from a
+ # submodule. This submodule path is hardcoded into their CMake definitions,
+ # and submodules are not included in their releases. Add a custom build step
+ # to download and extract the Protobufs.
+
+ # Adding such a step is rather complicated, so instead: create a separate
+ # ExternalProject that just fetches the Protobufs, then add a custom step
+ # to the main build to copy the Protobufs.
+ externalproject_add(opentelemetry_proto_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM}"
+ URL ${OPENTELEMETRY_PROTO_SOURCE_URL}
+ BUILD_COMMAND ""
+ CONFIGURE_COMMAND ""
+ INSTALL_COMMAND ""
+ EXCLUDE_FROM_ALL OFF)
+
+ add_dependencies(opentelemetry_dependencies nlohmann_json_ep opentelemetry_proto_ep)
+ if(gRPC_SOURCE STREQUAL "BUNDLED")
+ # TODO: opentelemetry-cpp::proto doesn't declare a dependency on gRPC, so
+ # even if we provide the location of gRPC, it'll fail to compile.
+ message(FATAL_ERROR "ARROW_WITH_OPENTELEMETRY cannot be configured with gRPC_SOURCE=BUNDLED. "
+ "See https://github.com/open-telemetry/opentelemetry-cpp/issues/1045"
+ )
+ endif()
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_API_ONLY=ON)
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR "|")
+ # JOIN is CMake >=3.12 only
+ string(REPLACE ";" "${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}"
+ OPENTELEMETRY_PREFIX_PATH "${OPENTELEMETRY_PREFIX_PATH_LIST}")
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ "-DCMAKE_PREFIX_PATH=${OPENTELEMETRY_PREFIX_PATH}")
+
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ # OpenTelemetry tries to determine the processor arch for vcpkg, which fails
+ # on s390x, even though it doesn't use vcpkg there. Tell it ARCH manually
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ endif()
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ externalproject_add_step(opentelemetry_ep download_proto
+ COMMAND ${CMAKE_COMMAND} -E copy_directory
+ $<TARGET_PROPERTY:opentelemetry_proto_ep,_EP_SOURCE_DIR>/opentelemetry
+ $<TARGET_PROPERTY:opentelemetry_ep,_EP_SOURCE_DIR>/third_party/opentelemetry-proto/opentelemetry
+ DEPENDEES download
+ DEPENDERS configure)
+ endif()
+
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ if(NOT ${CMAKE_VERSION} VERSION_LESS "3.15.0")
+ # Make sure all installed artifacts get cleaned up. Important for the RPM
+ # builds which fail if anything is left behind. Things in BUILD_BYPRODUCTS
+ # get cleaned automatically, but we can't list every header installed.
+ set_property(TARGET opentelemetry_ep
+ APPEND
+ PROPERTY ADDITIONAL_CLEAN_FILES "${OPENTELEMETRY_PREFIX}")
+ endif()
Review comment:
I think that we don't need this.
If we don't run any steps of `opentelemetry_ep` on `ninja install`, the RPM related problem can be solved. The `ninja install` in the RPM build is ran with `DESTDIR`. If the install step of `opentelemetry_ep` is ran with `ninja install`, OpenTelemetry related files are installed into `${DESTDIR}/....`. It causes the RPM related problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-971952058
Revision: 81d5fac3e1773db03060b5415c01e1ab5711309b
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1160](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1160)
|Task|Status|
|----|------|
|almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1160-github-almalinux-8-amd64)|
|centos-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1160-github-centos-8-amd64)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-971949016
@github-actions crossbow submit almalinux-8-amd64 centos-8-amd64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r736935906
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
+ public:
+ explicit OStreamJsonSpanExporter(std::ostream& sout = std::cerr) noexcept
+ : sout_(sout), shutdown_(false) {}
+ std::unique_ptr<sdktrace::Recordable> MakeRecordable() noexcept override {
+ return std::unique_ptr<sdktrace::Recordable>(new sdktrace::SpanData);
+ }
+ otel::sdk::common::ExportResult Export(
+ const nostd::span<std::unique_ptr<sdktrace::Recordable>>& spans) noexcept override {
+ if (shutdown_) return otel::sdk::common::ExportResult::kFailure;
+
+ for (auto& recordable : spans) {
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper> writer(sout_);
+ auto span = std::unique_ptr<sdktrace::SpanData>(
+ static_cast<sdktrace::SpanData*>(recordable.release()));
+ if (!span) continue;
+ char trace_id[32] = {0};
+ char span_id[16] = {0};
+ char parent_span_id[16] = {0};
+ span->GetTraceId().ToLowerBase16(trace_id);
+ span->GetSpanId().ToLowerBase16(span_id);
+ span->GetParentSpanId().ToLowerBase16(parent_span_id);
+
+ writer.StartObject();
+ writer.Key("name");
+ writer.String(span->GetName().data(), span->GetName().length());
+ writer.Key("trace_id");
+ writer.String(trace_id, 32);
+ writer.Key("span_id");
+ writer.String(span_id, 16);
+ writer.Key("parent_span_id");
+ writer.String(parent_span_id, 16);
+ writer.Key("start");
+ writer.Int64(span->GetStartTime().time_since_epoch().count());
+ writer.Key("duration");
+ writer.Int64(span->GetDuration().count());
+ writer.Key("description");
+ writer.String(span->GetDescription().data(), span->GetDescription().length());
+ writer.Key("kind");
+ writer.Int(static_cast<int>(span->GetSpanKind()));
+ writer.Key("status");
+ // TODO: this is expensive
+ writer.String(statuses_[static_cast<int>(span->GetStatus())]);
+ writer.Key("args");
+ writer.StartObject();
+ OwnedAttributeValueVisitor visitor(writer);
+ for (const auto& pair : span->GetAttributes()) {
+ writer.Key(pair.first.data(), pair.first.length());
+ nostd::visit(visitor, pair.second);
+ }
+ writer.EndObject();
+ writer.EndObject();
+ sout_.Put('\n');
+ }
+ sout_.Flush();
+ return otel::sdk::common::ExportResult::kSuccess;
+ }
+ bool Shutdown(std::chrono::microseconds) noexcept override {
+ shutdown_ = true;
+ return true;
+ }
+
+ private:
+ arrow::rapidjson::OStreamWrapper sout_;
+ bool shutdown_;
+ std::map<int, std::string> statuses_{{0, "Unset"}, {1, "Ok"}, {2, "Error"}};
+};
+#endif
+
+class ThreadIdSpanProcessor : public sdktrace::BatchSpanProcessor {
+ public:
+ using sdktrace::BatchSpanProcessor::BatchSpanProcessor;
+ void OnEnd(std::unique_ptr<sdktrace::Recordable>&& span) noexcept override {
+ std::stringstream thread_id;
+ thread_id << std::this_thread::get_id();
+ span->SetAttribute("thread_id", thread_id.str());
+ sdktrace::BatchSpanProcessor::OnEnd(std::move(span));
+ }
+};
+
+std::unique_ptr<sdktrace::SpanExporter> InitializeExporter() {
+ auto maybe_env_var = arrow::internal::GetEnvVar(kTracingBackendEnvVar);
+ if (maybe_env_var.ok()) {
+ auto env_var = maybe_env_var.ValueOrDie();
+ if (env_var == "json") {
+#ifdef ARROW_JSON
+ return std::unique_ptr<sdktrace::SpanExporter>(
+ new OStreamJsonSpanExporter(std::cerr));
+#else
+ ARROW_LOG(WARNING) << "Requested " << kTracingBackendEnvVar
+ << "=json but Arrow was built without ARROW_JSON";
+#endif
+ } else if (!env_var.empty()) {
+ ARROW_LOG(WARNING) << "Requested unknown backend " << kTracingBackendEnvVar << "="
+ << env_var;
+ }
+ }
+ return std::unique_ptr<sdktrace::SpanExporter>();
+}
+
+nostd::shared_ptr<sdktrace::TracerProvider> InitializeSdkTracerProvider() {
+ auto exporter = InitializeExporter();
+ if (exporter) {
+ sdktrace::BatchSpanProcessorOptions options;
+ options.max_queue_size = 16384;
+ options.schedule_delay_millis = std::chrono::milliseconds(500);
+ options.max_export_batch_size = 16384;
+ auto processor = std::unique_ptr<sdktrace::SpanProcessor>(
Review comment:
I can import arrow::util::make_unique
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r739914273
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
It's fine without, maybe we can get that packaged up before the next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-968116283
Most of the nightly failures look unrelated, however, the Almalinux/Amazon Linux/CentOS builds are failing because of "Installed but unpackaged file(s) found", so I need to fix that:
```
Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/arrow-7.0.0.dev103-1.el8.x86_64
error: Installed (but unpackaged) file(s) found:
RPM build errors:
/root/rpmbuild/BUILD/apache-arrow-7.0.0.dev103/cpp/build/opentelemetry_ep-install/include/opentelemetry/baggage/baggage.h
…snip…
```
(Autobrew and vcpkg are failing on default. Note that the vcpkg error looks like the one we saw during 6.0.0 verification: I would guess we're both building vcpkg and installing it as a dependency, and then using the headers of one but linking against the other.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-974112875
It looks like things are generally passing here - any objections?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r753524100
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3865,6 +3902,212 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api ext sdk)
+ set(_OPENTELEMETRY_LIBS
+ common
+ http_client_curl
+ ostream_span_exporter
+ otlp_http_exporter
+ otlp_recordable
+ proto
+ resources
+ trace
+ version)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ if(_OPENTELEMETRY_LIB STREQUAL "http_client_curl")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "ostream_span_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_ostream_span${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "otlp_http_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_otlp_http${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ else()
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ endif()
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST)
+ # Don't specify the DEPENDS unless we actually have dependencies, else
+ # Ninja/other build systems may consider this target to always be dirty
+ set(_OPENTELEMETRY_DEPENDENCIES)
+ add_custom_target(opentelemetry_dependencies)
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ set(_OPENTELEMETRY_DEPENDENCIES "opentelemetry_dependencies")
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ list(APPEND OPENTELEMETRY_PREFIX_PATH_LIST ${NLOHMANN_JSON_PREFIX})
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_OTLP=ON
+ -DWITH_OTLP_HTTP=ON -DWITH_OTLP_GRPC=OFF)
+
+ # OpenTelemetry with OTLP enabled requires Protobuf definitions from a
+ # submodule. This submodule path is hardcoded into their CMake definitions,
+ # and submodules are not included in their releases. Add a custom build step
+ # to download and extract the Protobufs.
+
+ # Adding such a step is rather complicated, so instead: create a separate
+ # ExternalProject that just fetches the Protobufs, then add a custom step
+ # to the main build to copy the Protobufs.
+ externalproject_add(opentelemetry_proto_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM}"
+ URL ${OPENTELEMETRY_PROTO_SOURCE_URL}
+ BUILD_COMMAND ""
+ CONFIGURE_COMMAND ""
+ INSTALL_COMMAND ""
+ EXCLUDE_FROM_ALL OFF)
+
+ add_dependencies(opentelemetry_dependencies nlohmann_json_ep opentelemetry_proto_ep)
+ if(gRPC_SOURCE STREQUAL "BUNDLED")
+ # TODO: opentelemetry-cpp::proto doesn't declare a dependency on gRPC, so
+ # even if we provide the location of gRPC, it'll fail to compile.
+ message(FATAL_ERROR "ARROW_WITH_OPENTELEMETRY cannot be configured with gRPC_SOURCE=BUNDLED. "
+ "See https://github.com/open-telemetry/opentelemetry-cpp/issues/1045"
+ )
+ endif()
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_API_ONLY=ON)
Review comment:
Sorry, I didn't fully change OpenTelemetry to be a completely optional component. I've reorganized things such that there's only a singular ARROW_WITH_OPENTELEMETRY check. As a consequence this branch is no longer needed in the first place.
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3865,6 +3902,212 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api ext sdk)
+ set(_OPENTELEMETRY_LIBS
+ common
+ http_client_curl
+ ostream_span_exporter
+ otlp_http_exporter
+ otlp_recordable
+ proto
+ resources
+ trace
+ version)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ if(_OPENTELEMETRY_LIB STREQUAL "http_client_curl")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "ostream_span_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_ostream_span${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "otlp_http_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_otlp_http${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ else()
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ endif()
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST)
+ # Don't specify the DEPENDS unless we actually have dependencies, else
+ # Ninja/other build systems may consider this target to always be dirty
+ set(_OPENTELEMETRY_DEPENDENCIES)
+ add_custom_target(opentelemetry_dependencies)
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ set(_OPENTELEMETRY_DEPENDENCIES "opentelemetry_dependencies")
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ list(APPEND OPENTELEMETRY_PREFIX_PATH_LIST ${NLOHMANN_JSON_PREFIX})
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_OTLP=ON
+ -DWITH_OTLP_HTTP=ON -DWITH_OTLP_GRPC=OFF)
+
+ # OpenTelemetry with OTLP enabled requires Protobuf definitions from a
+ # submodule. This submodule path is hardcoded into their CMake definitions,
+ # and submodules are not included in their releases. Add a custom build step
+ # to download and extract the Protobufs.
+
+ # Adding such a step is rather complicated, so instead: create a separate
+ # ExternalProject that just fetches the Protobufs, then add a custom step
+ # to the main build to copy the Protobufs.
+ externalproject_add(opentelemetry_proto_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM}"
+ URL ${OPENTELEMETRY_PROTO_SOURCE_URL}
+ BUILD_COMMAND ""
+ CONFIGURE_COMMAND ""
+ INSTALL_COMMAND ""
+ EXCLUDE_FROM_ALL OFF)
+
+ add_dependencies(opentelemetry_dependencies nlohmann_json_ep opentelemetry_proto_ep)
+ if(gRPC_SOURCE STREQUAL "BUNDLED")
+ # TODO: opentelemetry-cpp::proto doesn't declare a dependency on gRPC, so
+ # even if we provide the location of gRPC, it'll fail to compile.
+ message(FATAL_ERROR "ARROW_WITH_OPENTELEMETRY cannot be configured with gRPC_SOURCE=BUNDLED. "
+ "See https://github.com/open-telemetry/opentelemetry-cpp/issues/1045"
+ )
+ endif()
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_API_ONLY=ON)
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR "|")
+ # JOIN is CMake >=3.12 only
+ string(REPLACE ";" "${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}"
+ OPENTELEMETRY_PREFIX_PATH "${OPENTELEMETRY_PREFIX_PATH_LIST}")
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ "-DCMAKE_PREFIX_PATH=${OPENTELEMETRY_PREFIX_PATH}")
+
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ # OpenTelemetry tries to determine the processor arch for vcpkg, which fails
+ # on s390x, even though it doesn't use vcpkg there. Tell it ARCH manually
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ endif()
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ externalproject_add_step(opentelemetry_ep download_proto
+ COMMAND ${CMAKE_COMMAND} -E copy_directory
+ $<TARGET_PROPERTY:opentelemetry_proto_ep,_EP_SOURCE_DIR>/opentelemetry
+ $<TARGET_PROPERTY:opentelemetry_ep,_EP_SOURCE_DIR>/third_party/opentelemetry-proto/opentelemetry
+ DEPENDEES download
+ DEPENDERS configure)
+ endif()
+
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+ set_target_properties(opentelemetry-cpp::http_client_curl
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::ext;CURL::libcurl")
+ set_target_properties(opentelemetry-cpp::proto
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "${ARROW_PROTOBUF_LIBPROTOBUF}")
+ set_target_properties(opentelemetry-cpp::otlp_recordable
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::trace;opentelemetry-cpp::resources;opentelemetry-cpp::proto"
+ )
+ set_target_properties(opentelemetry-cpp::otlp_http_exporter
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::otlp_recordable;opentelemetry-cpp::http_client_curl;nlohmann_json::nlohmann_json"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+if(ARROW_WITH_OPENTELEMETRY)
+ set(opentelemetry-cpp_SOURCE "AUTO")
+ if(ARROW_WITH_OPENTELEMETRY)
+ build_nlohmann_json_once()
+ find_curl()
Review comment:
Moved to the top of the build macro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-970588900
Revision: a9cb2fc1beb705ec275054025419ab94608ed51a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1149](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1149)
|Task|Status|
|----|------|
|almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1149-github-almalinux-8-amd64)|
|centos-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1149-github-centos-8-amd64)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r736934319
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
The motivation here was to have a convenient way to just get a log file that we could then parse/analyze with something else. Hence:
1) Yes we are a library, but for development purposes it's easier to just have very basic configuration built in
2) On top of that, [the C++ and Python libraries do not interoperate](https://github.com/open-telemetry/community/discussions/734) which makes it rather inconvenient to use with PyArrow
3) The output of OStreamSpanExporter is very annoying to parse compared to JSON
That said, the collector seems like it should mostly address all this. IIRC, I had some issues with getting it to point to our gRPC build and such and decided to just drop it for expediency but we should figure it out instead of doing something special.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r736935583
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
+ public:
+ explicit OStreamJsonSpanExporter(std::ostream& sout = std::cerr) noexcept
+ : sout_(sout), shutdown_(false) {}
+ std::unique_ptr<sdktrace::Recordable> MakeRecordable() noexcept override {
+ return std::unique_ptr<sdktrace::Recordable>(new sdktrace::SpanData);
+ }
+ otel::sdk::common::ExportResult Export(
+ const nostd::span<std::unique_ptr<sdktrace::Recordable>>& spans) noexcept override {
+ if (shutdown_) return otel::sdk::common::ExportResult::kFailure;
+
+ for (auto& recordable : spans) {
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper> writer(sout_);
+ auto span = std::unique_ptr<sdktrace::SpanData>(
+ static_cast<sdktrace::SpanData*>(recordable.release()));
+ if (!span) continue;
+ char trace_id[32] = {0};
+ char span_id[16] = {0};
+ char parent_span_id[16] = {0};
+ span->GetTraceId().ToLowerBase16(trace_id);
+ span->GetSpanId().ToLowerBase16(span_id);
+ span->GetParentSpanId().ToLowerBase16(parent_span_id);
+
+ writer.StartObject();
+ writer.Key("name");
+ writer.String(span->GetName().data(), span->GetName().length());
+ writer.Key("trace_id");
+ writer.String(trace_id, 32);
+ writer.Key("span_id");
+ writer.String(span_id, 16);
+ writer.Key("parent_span_id");
+ writer.String(parent_span_id, 16);
+ writer.Key("start");
+ writer.Int64(span->GetStartTime().time_since_epoch().count());
+ writer.Key("duration");
+ writer.Int64(span->GetDuration().count());
+ writer.Key("description");
+ writer.String(span->GetDescription().data(), span->GetDescription().length());
+ writer.Key("kind");
+ writer.Int(static_cast<int>(span->GetSpanKind()));
+ writer.Key("status");
+ // TODO: this is expensive
+ writer.String(statuses_[static_cast<int>(span->GetStatus())]);
+ writer.Key("args");
+ writer.StartObject();
+ OwnedAttributeValueVisitor visitor(writer);
+ for (const auto& pair : span->GetAttributes()) {
+ writer.Key(pair.first.data(), pair.first.length());
+ nostd::visit(visitor, pair.second);
+ }
+ writer.EndObject();
+ writer.EndObject();
+ sout_.Put('\n');
+ }
+ sout_.Flush();
+ return otel::sdk::common::ExportResult::kSuccess;
+ }
+ bool Shutdown(std::chrono::microseconds) noexcept override {
+ shutdown_ = true;
+ return true;
+ }
+
+ private:
+ arrow::rapidjson::OStreamWrapper sout_;
+ bool shutdown_;
+ std::map<int, std::string> statuses_{{0, "Unset"}, {1, "Ok"}, {2, "Error"}};
+};
+#endif
+
+class ThreadIdSpanProcessor : public sdktrace::BatchSpanProcessor {
Review comment:
I _think_ when I looked at it, it would be the thread of the span (the processor is called inline instead of on the background thread), and that is the intent here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r738724692
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
Ok, so what I think I'll do here is remove the formatter and include a way to configure the ostream and OTLP exporters. I want to still _optionally_ include these exporters, for cases like PyArrow and R (which can't realistically link to OTel and configure it) and just to make like easier for Arrow developers. But anything we ship will only use the OpenTelemetry API.
I agree libraries should only use the API but Arrow is in a weird spot. If the Python OTel bindings interoperated with the C++ ones (and we had R bindings) then the issue here would be moot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r738754458
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
I would guess that most Python libraries with exporters would be using the Python library, but yeah - we should ship Arrow without the exporter by default. (I think it should be possible to eventually ship a second package with only the exporters, perhaps? At least on Conda, maybe not as a wheel.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r753534661
##########
File path: cpp/cmake_modules/DefineOptions.cmake
##########
@@ -371,6 +371,9 @@ if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
define_option(ARROW_WITH_BACKTRACE "Build with backtrace support" ON)
+ define_option(ARROW_WITH_OPENTELEMETRY
+ "Build libraries with OpenTelemetry support for distributed tracing" OFF)
Review comment:
I see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956334909
> Example of which part in particular? For this PR, you can just try checking it out and building with `-DARROW_WITH_OPENTELEMETRY=ON`
An example of a client application that would hit an instrumented Arrow API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956333167
Example of which part in particular? For this PR, you can just try checking it out and building with `-DARROW_WITH_OPENTELEMETRY=ON`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r739898822
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
I can, though without packages in Conda etc. it'll be a little more annoying to test things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956330534
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956500244
Now this can build against a system-installed OTel, and you can enable the OTLP HTTP exporter (I checked that this works for me, thanks for the pointers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r738742092
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
True. Sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r739891424
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
Any chance you can do the same thing that's done elsewhere in this file and call `find_package` if `OpenTelemetry_SOURCE == "SYSTEM"`?
Bundling unconditionally like this makes it really difficult to deal with OpenTelemetry provided by a package manager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-967349975
Tests are passing here, so we just need a reviewer, we should check that nightlies pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-970669221
Revision: a9cb2fc1beb705ec275054025419ab94608ed51a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1150](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1150)
|Task|Status|
|----|------|
|almalinux-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-almalinux-8-amd64)|
|almalinux-8-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|amazon-linux-2-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-amazon-linux-2-amd64)|
|centos-7-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-centos-7-amd64)|
|centos-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-centos-8-amd64)|
|centos-8-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|conda-clean|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-clean)|
|conda-linux-gcc-py310-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py310-arm64)|
|conda-linux-gcc-py310-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py310-cuda)|
|conda-linux-gcc-py36-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py36-arm64)|
|conda-linux-gcc-py36-cpu-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py36-cpu-r40)|
|conda-linux-gcc-py36-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py36-cuda)|
|conda-linux-gcc-py37-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py37-arm64)|
|conda-linux-gcc-py37-cpu-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py37-cpu-r41)|
|conda-linux-gcc-py37-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py37-cuda)|
|conda-linux-gcc-py38-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py38-arm64)|
|conda-linux-gcc-py38-cpu|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py38-cpu)|
|conda-linux-gcc-py38-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py38-cuda)|
|conda-linux-gcc-py39-arm64|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py39-arm64)|
|conda-linux-gcc-py39-cpu|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py39-cpu)|
|conda-linux-gcc-py39-cuda|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-linux-gcc-py39-cuda)|
|conda-osx-arm64-clang-py38|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-osx-arm64-clang-py38)|
|conda-osx-arm64-clang-py39|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-osx-arm64-clang-py39)|
|conda-osx-clang-py36-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-osx-clang-py36-r40)|
|conda-osx-clang-py37-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-osx-clang-py37-r41)|
|conda-osx-clang-py38|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-osx-clang-py38)|
|conda-osx-clang-py39|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-osx-clang-py39)|
|conda-win-vs2017-py36-r40|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-win-vs2017-py36-r40)|
|conda-win-vs2017-py37-r41|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-win-vs2017-py37-r41)|
|conda-win-vs2017-py38|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-win-vs2017-py38)|
|conda-win-vs2017-py39|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-conda-win-vs2017-py39)|
|debian-bookworm-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-debian-bookworm-amd64)|
|debian-bookworm-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|debian-bullseye-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-debian-bullseye-amd64)|
|debian-bullseye-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|debian-buster-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-debian-buster-amd64)|
|debian-buster-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|example-cpp-minimal-build-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-example-cpp-minimal-build-static)|
|example-cpp-minimal-build-static-system-dependency|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-example-cpp-minimal-build-static-system-dependency)|
|homebrew-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-homebrew-cpp)|
|homebrew-r-autobrew|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-homebrew-r-autobrew)|
|java-jars|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-java-jars)|
|nuget|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-nuget)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-python-sdist)|
|test-build-cpp-fuzz|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-build-cpp-fuzz)|
|test-build-vcpkg-win|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-build-vcpkg-win)|
|test-conda-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-cpp)|
|test-conda-cpp-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-conda-cpp-valgrind)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.10)|
|test-conda-python-3.6|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.6)|
|test-conda-python-3.6-pandas-0.23|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.6-pandas-0.23)|
|test-conda-python-3.7|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.7)|
|test-conda-python-3.7-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.7-hdfs-2.9.2)|
|test-conda-python-3.7-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.7-hdfs-3.2.1)|
|test-conda-python-3.7-kartothek-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.7-kartothek-latest)|
|test-conda-python-3.7-kartothek-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.7-kartothek-master)|
|test-conda-python-3.7-pandas-0.24|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.7-pandas-0.24)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.7-pandas-latest)|
|test-conda-python-3.7-spark-v3.1.2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.7-spark-v3.1.2)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.8)|
|test-conda-python-3.8-hypothesis|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.8-hypothesis)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.8-pandas-latest)|
|test-conda-python-3.8-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.8-pandas-nightly)|
|test-conda-python-3.8-spark-v3.2.0|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.8-spark-v3.2.0)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.9)|
|test-conda-python-3.9-dask-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.9-dask-latest)|
|test-conda-python-3.9-dask-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.9-dask-master)|
|test-conda-python-3.9-pandas-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.9-pandas-master)|
|test-conda-python-3.9-spark-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-conda-python-3.9-spark-master)|
|test-debian-10-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-debian-10-cpp)|
|test-debian-11-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-debian-11-cpp)|
|test-debian-11-go-1.15|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-debian-11-go-1.15)|
|test-debian-11-python-3|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-debian-11-python-3)|
|test-debian-c-glib|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-debian-c-glib)|
|test-debian-ruby|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-debian-ruby)|
|test-fedora-33-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-fedora-33-cpp)|
|test-fedora-33-python-3|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-fedora-33-python-3)|
|test-fedora-r-clang-sanitizer|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-fedora-r-clang-sanitizer)|
|test-r-arrow-backwards-compatibility|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-arrow-backwards-compatibility)|
|test-r-depsource-auto|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-depsource-auto)|
|test-r-depsource-system|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-depsource-system)|
|test-r-devdocs|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-devdocs)|
|test-r-gcc-11|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-gcc-11)|
|test-r-install-local|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-install-local)|
|test-r-linux-as-cran|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-linux-as-cran)|
|test-r-linux-rchk|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-linux-rchk)|
|test-r-linux-valgrind|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-linux-valgrind)|
|test-r-minimal-build|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-minimal-build)|
|test-r-offline-maximal|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-offline-maximal)|
|test-r-offline-minimal|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-offline-minimal)|
|test-r-rhub-debian-gcc-devel-lto-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-rhub-debian-gcc-devel-lto-latest)|
|test-r-rhub-ubuntu-gcc-release-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-rhub-ubuntu-gcc-release-latest)|
|test-r-rocker-r-base-latest|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-rocker-r-base-latest)|
|test-r-rstudio-r-base-3.6-bionic|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-rstudio-r-base-3.6-bionic)|
|test-r-rstudio-r-base-3.6-centos7-devtoolset-8|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-rstudio-r-base-3.6-centos7-devtoolset-8)|
|test-r-rstudio-r-base-3.6-centos8|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-rstudio-r-base-3.6-centos8)|
|test-r-rstudio-r-base-3.6-opensuse15|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-rstudio-r-base-3.6-opensuse15)|
|test-r-rstudio-r-base-3.6-opensuse42|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-r-rstudio-r-base-3.6-opensuse42)|
|test-r-ubuntu-21.04|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-ubuntu-21.04)|
|test-r-versions|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-r-versions)|
|test-skyhook-integration|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-skyhook-integration)|
|test-ubuntu-18.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-18.04-cpp)|
|test-ubuntu-18.04-cpp-release|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-18.04-cpp-release)|
|test-ubuntu-18.04-cpp-static|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-18.04-cpp-static)|
|test-ubuntu-18.04-python-3|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-ubuntu-18.04-python-3)|
|test-ubuntu-18.04-r-sanitizer|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-ubuntu-18.04-r-sanitizer)|
|test-ubuntu-20.04-cpp|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-20.04-cpp)|
|test-ubuntu-20.04-cpp-14|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-20.04-cpp-14)|
|test-ubuntu-20.04-cpp-17|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-20.04-cpp-17)|
|test-ubuntu-20.04-cpp-bundled|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-20.04-cpp-bundled)|
|test-ubuntu-20.04-cpp-thread-sanitizer|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-20.04-cpp-thread-sanitizer)|
|test-ubuntu-20.10-docs|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-ubuntu-20.10-docs)|
|test-ubuntu-c-glib|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-c-glib)|
|test-ubuntu-default-docs|[](https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-1150-azure-test-ubuntu-default-docs)|
|test-ubuntu-ruby|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-test-ubuntu-ruby)|
|ubuntu-bionic-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-ubuntu-bionic-amd64)|
|ubuntu-bionic-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|ubuntu-focal-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-ubuntu-focal-amd64)|
|ubuntu-focal-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|ubuntu-hirsute-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-ubuntu-hirsute-amd64)|
|ubuntu-hirsute-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|ubuntu-impish-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-ubuntu-impish-amd64)|
|ubuntu-impish-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-macos-big-sur-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-big-sur-cp310-arm64)|
|wheel-macos-big-sur-cp310-universal2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-big-sur-cp310-universal2)|
|wheel-macos-big-sur-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-big-sur-cp38-arm64)|
|wheel-macos-big-sur-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-big-sur-cp39-arm64)|
|wheel-macos-big-sur-cp39-universal2|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-big-sur-cp39-universal2)|
|wheel-macos-high-sierra-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-high-sierra-cp310-amd64)|
|wheel-macos-high-sierra-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-high-sierra-cp36-amd64)|
|wheel-macos-high-sierra-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-high-sierra-cp37-amd64)|
|wheel-macos-high-sierra-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-high-sierra-cp38-amd64)|
|wheel-macos-high-sierra-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-high-sierra-cp39-amd64)|
|wheel-macos-mavericks-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-mavericks-cp310-amd64)|
|wheel-macos-mavericks-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-mavericks-cp36-amd64)|
|wheel-macos-mavericks-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-mavericks-cp37-amd64)|
|wheel-macos-mavericks-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-mavericks-cp38-amd64)|
|wheel-macos-mavericks-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-macos-mavericks-cp39-amd64)|
|wheel-manylinux2010-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2010-cp310-amd64)|
|wheel-manylinux2010-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2010-cp36-amd64)|
|wheel-manylinux2010-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2010-cp37-amd64)|
|wheel-manylinux2010-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2010-cp38-amd64)|
|wheel-manylinux2010-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2010-cp39-amd64)|
|wheel-manylinux2014-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2014-cp310-amd64)|
|wheel-manylinux2014-cp310-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2014-cp36-amd64)|
|wheel-manylinux2014-cp36-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2014-cp37-amd64)|
|wheel-manylinux2014-cp37-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2014-cp38-amd64)|
|wheel-manylinux2014-cp38-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-manylinux2014-cp39-amd64)|
|wheel-manylinux2014-cp39-arm64|[](https://travis-ci.com/ursacomputing/crossbow/branches)|
|wheel-windows-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-windows-cp310-amd64)|
|wheel-windows-cp36-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-windows-cp36-amd64)|
|wheel-windows-cp37-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-windows-cp37-amd64)|
|wheel-windows-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-windows-cp38-amd64)|
|wheel-windows-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-1150-github-wheel-windows-cp39-amd64)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r749707000
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
This is a failure on my part for not communicating well, sorry.
A few of us working on the C++ query engine/datasets project have been using OpenTelemetry more recently to do some profiling and performance analysis, and so I reoriented the PR along those lines, instead of focusing on Flight. We'll try to get a mailing list post up to explain the thinking before pushing on this more.
The other reason is that I found it difficult to #ifdef OpenTelemetry out completely in those use cases, since fully using it requires you to occasionally pass around context objects and so some internal function signatures changed. Hence it was easier to just enable it unconditionally (but in "api-only" mode). However we may be able to use it without going that far, so for this PR I will try to change it to be conditional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-970587638
@github-actions crossbow -g submit almalinux-8-amd64 centos-8-amd64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r736936788
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
+ public:
+ explicit OStreamJsonSpanExporter(std::ostream& sout = std::cerr) noexcept
+ : sout_(sout), shutdown_(false) {}
+ std::unique_ptr<sdktrace::Recordable> MakeRecordable() noexcept override {
+ return std::unique_ptr<sdktrace::Recordable>(new sdktrace::SpanData);
+ }
+ otel::sdk::common::ExportResult Export(
+ const nostd::span<std::unique_ptr<sdktrace::Recordable>>& spans) noexcept override {
+ if (shutdown_) return otel::sdk::common::ExportResult::kFailure;
+
+ for (auto& recordable : spans) {
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper> writer(sout_);
+ auto span = std::unique_ptr<sdktrace::SpanData>(
+ static_cast<sdktrace::SpanData*>(recordable.release()));
+ if (!span) continue;
+ char trace_id[32] = {0};
+ char span_id[16] = {0};
+ char parent_span_id[16] = {0};
+ span->GetTraceId().ToLowerBase16(trace_id);
+ span->GetSpanId().ToLowerBase16(span_id);
+ span->GetParentSpanId().ToLowerBase16(parent_span_id);
+
+ writer.StartObject();
+ writer.Key("name");
+ writer.String(span->GetName().data(), span->GetName().length());
+ writer.Key("trace_id");
+ writer.String(trace_id, 32);
+ writer.Key("span_id");
+ writer.String(span_id, 16);
+ writer.Key("parent_span_id");
+ writer.String(parent_span_id, 16);
+ writer.Key("start");
+ writer.Int64(span->GetStartTime().time_since_epoch().count());
+ writer.Key("duration");
+ writer.Int64(span->GetDuration().count());
+ writer.Key("description");
+ writer.String(span->GetDescription().data(), span->GetDescription().length());
+ writer.Key("kind");
+ writer.Int(static_cast<int>(span->GetSpanKind()));
+ writer.Key("status");
+ // TODO: this is expensive
+ writer.String(statuses_[static_cast<int>(span->GetStatus())]);
+ writer.Key("args");
+ writer.StartObject();
+ OwnedAttributeValueVisitor visitor(writer);
+ for (const auto& pair : span->GetAttributes()) {
+ writer.Key(pair.first.data(), pair.first.length());
+ nostd::visit(visitor, pair.second);
+ }
+ writer.EndObject();
+ writer.EndObject();
+ sout_.Put('\n');
+ }
+ sout_.Flush();
+ return otel::sdk::common::ExportResult::kSuccess;
+ }
+ bool Shutdown(std::chrono::microseconds) noexcept override {
+ shutdown_ = true;
+ return true;
+ }
+
+ private:
+ arrow::rapidjson::OStreamWrapper sout_;
+ bool shutdown_;
+ std::map<int, std::string> statuses_{{0, "Unset"}, {1, "Ok"}, {2, "Error"}};
+};
+#endif
+
+class ThreadIdSpanProcessor : public sdktrace::BatchSpanProcessor {
+ public:
+ using sdktrace::BatchSpanProcessor::BatchSpanProcessor;
+ void OnEnd(std::unique_ptr<sdktrace::Recordable>&& span) noexcept override {
+ std::stringstream thread_id;
+ thread_id << std::this_thread::get_id();
+ span->SetAttribute("thread_id", thread_id.str());
+ sdktrace::BatchSpanProcessor::OnEnd(std::move(span));
+ }
+};
+
+std::unique_ptr<sdktrace::SpanExporter> InitializeExporter() {
+ auto maybe_env_var = arrow::internal::GetEnvVar(kTracingBackendEnvVar);
+ if (maybe_env_var.ok()) {
+ auto env_var = maybe_env_var.ValueOrDie();
+ if (env_var == "json") {
+#ifdef ARROW_JSON
+ return std::unique_ptr<sdktrace::SpanExporter>(
+ new OStreamJsonSpanExporter(std::cerr));
+#else
+ ARROW_LOG(WARNING) << "Requested " << kTracingBackendEnvVar
+ << "=json but Arrow was built without ARROW_JSON";
+#endif
+ } else if (!env_var.empty()) {
+ ARROW_LOG(WARNING) << "Requested unknown backend " << kTracingBackendEnvVar << "="
+ << env_var;
+ }
+ }
+ return std::unique_ptr<sdktrace::SpanExporter>();
+}
+
+nostd::shared_ptr<sdktrace::TracerProvider> InitializeSdkTracerProvider() {
+ auto exporter = InitializeExporter();
+ if (exporter) {
+ sdktrace::BatchSpanProcessorOptions options;
+ options.max_queue_size = 16384;
+ options.schedule_delay_millis = std::chrono::milliseconds(500);
+ options.max_export_batch_size = 16384;
+ auto processor = std::unique_ptr<sdktrace::SpanProcessor>(
+ new ThreadIdSpanProcessor(std::move(exporter), options));
+ return nostd::shared_ptr<sdktrace::TracerProvider>(
Review comment:
I don't remember if there was a reason why. I can go fiddle with it. (I think there's no nostd::make_shared?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-984014058
The fix https://github.com/open-telemetry/opentelemetry-cpp/commit/eca856df5c2f3e73bd57a7d84d7476ca62f40e80 is in v1.1.0 so we should be able to remove the gRPC dependency in ARROW-14957.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10260: ARROW-12671: [C++][Python][FlightRPC] WIP: Simple OpenTelemetry integration
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-833754260
https://issues.apache.org/jira/browse/ARROW-12671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r749766465
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
Oh, sorry. I missed the post.
> Maybe then we could enable OT only if "flight" or "query engine" were enabled? Either way, I agree this is a good discussion for the ML.
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-969313047
> Most of the nightly failures look unrelated, however, the Almalinux/Amazon Linux/CentOS builds are failing because of "Installed but unpackaged file(s) found", so I need to fix that:
>
> ```
> Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/arrow-7.0.0.dev103-1.el8.x86_64
> error: Installed (but unpackaged) file(s) found:
>
>
> RPM build errors:
> /root/rpmbuild/BUILD/apache-arrow-7.0.0.dev103/cpp/build/opentelemetry_ep-install/include/opentelemetry/baggage/baggage.h
> …snip…
> ```
>
> (Autobrew and vcpkg are failing on default. Note that the vcpkg error looks like the one we saw during 6.0.0 verification: I would guess we're both building vcpkg and installing it as a dependency, and then using the headers of one but linking against the other.)
Sorry for the trouble @kou, but do you know what's going on with the RPM build error? It appears `opentelemetry_ep-install` gets left behind in `/root/rpmbuild/BUILDROOT/arrow-7.0.0-0.dev20211101.el8.x86_64/root/rpmbuild/BUILD/apache-arrow-7.0.0/` and so rpmbuild complains, but I can't figure out what is responsible for cleaning up the rest of the build files (`ninja clean` on my local machine doesn't actually delete them)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r753517666
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3865,6 +3902,212 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api ext sdk)
+ set(_OPENTELEMETRY_LIBS
+ common
+ http_client_curl
+ ostream_span_exporter
+ otlp_http_exporter
+ otlp_recordable
+ proto
+ resources
+ trace
+ version)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ if(_OPENTELEMETRY_LIB STREQUAL "http_client_curl")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "ostream_span_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_ostream_span${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "otlp_http_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_otlp_http${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ else()
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ endif()
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST)
+ # Don't specify the DEPENDS unless we actually have dependencies, else
+ # Ninja/other build systems may consider this target to always be dirty
+ set(_OPENTELEMETRY_DEPENDENCIES)
+ add_custom_target(opentelemetry_dependencies)
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ set(_OPENTELEMETRY_DEPENDENCIES "opentelemetry_dependencies")
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ list(APPEND OPENTELEMETRY_PREFIX_PATH_LIST ${NLOHMANN_JSON_PREFIX})
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_OTLP=ON
+ -DWITH_OTLP_HTTP=ON -DWITH_OTLP_GRPC=OFF)
+
+ # OpenTelemetry with OTLP enabled requires Protobuf definitions from a
+ # submodule. This submodule path is hardcoded into their CMake definitions,
+ # and submodules are not included in their releases. Add a custom build step
+ # to download and extract the Protobufs.
+
+ # Adding such a step is rather complicated, so instead: create a separate
+ # ExternalProject that just fetches the Protobufs, then add a custom step
+ # to the main build to copy the Protobufs.
+ externalproject_add(opentelemetry_proto_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM}"
+ URL ${OPENTELEMETRY_PROTO_SOURCE_URL}
+ BUILD_COMMAND ""
+ CONFIGURE_COMMAND ""
+ INSTALL_COMMAND ""
+ EXCLUDE_FROM_ALL OFF)
+
+ add_dependencies(opentelemetry_dependencies nlohmann_json_ep opentelemetry_proto_ep)
+ if(gRPC_SOURCE STREQUAL "BUNDLED")
+ # TODO: opentelemetry-cpp::proto doesn't declare a dependency on gRPC, so
+ # even if we provide the location of gRPC, it'll fail to compile.
+ message(FATAL_ERROR "ARROW_WITH_OPENTELEMETRY cannot be configured with gRPC_SOURCE=BUNDLED. "
+ "See https://github.com/open-telemetry/opentelemetry-cpp/issues/1045"
+ )
+ endif()
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_API_ONLY=ON)
Review comment:
Could you use `list(APPEND ...)`?
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3865,6 +3902,212 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api ext sdk)
+ set(_OPENTELEMETRY_LIBS
+ common
+ http_client_curl
+ ostream_span_exporter
+ otlp_http_exporter
+ otlp_recordable
+ proto
+ resources
+ trace
+ version)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ if(_OPENTELEMETRY_LIB STREQUAL "http_client_curl")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "ostream_span_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_ostream_span${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "otlp_http_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_otlp_http${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ else()
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ endif()
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST)
+ # Don't specify the DEPENDS unless we actually have dependencies, else
+ # Ninja/other build systems may consider this target to always be dirty
+ set(_OPENTELEMETRY_DEPENDENCIES)
+ add_custom_target(opentelemetry_dependencies)
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ set(_OPENTELEMETRY_DEPENDENCIES "opentelemetry_dependencies")
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ list(APPEND OPENTELEMETRY_PREFIX_PATH_LIST ${NLOHMANN_JSON_PREFIX})
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_OTLP=ON
+ -DWITH_OTLP_HTTP=ON -DWITH_OTLP_GRPC=OFF)
+
+ # OpenTelemetry with OTLP enabled requires Protobuf definitions from a
+ # submodule. This submodule path is hardcoded into their CMake definitions,
+ # and submodules are not included in their releases. Add a custom build step
+ # to download and extract the Protobufs.
+
+ # Adding such a step is rather complicated, so instead: create a separate
+ # ExternalProject that just fetches the Protobufs, then add a custom step
+ # to the main build to copy the Protobufs.
+ externalproject_add(opentelemetry_proto_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM}"
+ URL ${OPENTELEMETRY_PROTO_SOURCE_URL}
+ BUILD_COMMAND ""
+ CONFIGURE_COMMAND ""
+ INSTALL_COMMAND ""
+ EXCLUDE_FROM_ALL OFF)
+
+ add_dependencies(opentelemetry_dependencies nlohmann_json_ep opentelemetry_proto_ep)
+ if(gRPC_SOURCE STREQUAL "BUNDLED")
+ # TODO: opentelemetry-cpp::proto doesn't declare a dependency on gRPC, so
+ # even if we provide the location of gRPC, it'll fail to compile.
+ message(FATAL_ERROR "ARROW_WITH_OPENTELEMETRY cannot be configured with gRPC_SOURCE=BUNDLED. "
+ "See https://github.com/open-telemetry/opentelemetry-cpp/issues/1045"
+ )
+ endif()
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_API_ONLY=ON)
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR "|")
+ # JOIN is CMake >=3.12 only
+ string(REPLACE ";" "${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}"
+ OPENTELEMETRY_PREFIX_PATH "${OPENTELEMETRY_PREFIX_PATH_LIST}")
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ "-DCMAKE_PREFIX_PATH=${OPENTELEMETRY_PREFIX_PATH}")
+
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ # OpenTelemetry tries to determine the processor arch for vcpkg, which fails
+ # on s390x, even though it doesn't use vcpkg there. Tell it ARCH manually
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ endif()
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ externalproject_add_step(opentelemetry_ep download_proto
+ COMMAND ${CMAKE_COMMAND} -E copy_directory
+ $<TARGET_PROPERTY:opentelemetry_proto_ep,_EP_SOURCE_DIR>/opentelemetry
+ $<TARGET_PROPERTY:opentelemetry_ep,_EP_SOURCE_DIR>/third_party/opentelemetry-proto/opentelemetry
+ DEPENDEES download
+ DEPENDERS configure)
+ endif()
+
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+ set_target_properties(opentelemetry-cpp::http_client_curl
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::ext;CURL::libcurl")
+ set_target_properties(opentelemetry-cpp::proto
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "${ARROW_PROTOBUF_LIBPROTOBUF}")
+ set_target_properties(opentelemetry-cpp::otlp_recordable
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::trace;opentelemetry-cpp::resources;opentelemetry-cpp::proto"
+ )
+ set_target_properties(opentelemetry-cpp::otlp_http_exporter
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::otlp_recordable;opentelemetry-cpp::http_client_curl;nlohmann_json::nlohmann_json"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+if(ARROW_WITH_OPENTELEMETRY)
+ set(opentelemetry-cpp_SOURCE "AUTO")
+ if(ARROW_WITH_OPENTELEMETRY)
Review comment:
This is redundant.
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3865,6 +3902,212 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api ext sdk)
+ set(_OPENTELEMETRY_LIBS
+ common
+ http_client_curl
+ ostream_span_exporter
+ otlp_http_exporter
+ otlp_recordable
+ proto
+ resources
+ trace
+ version)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ if(_OPENTELEMETRY_LIB STREQUAL "http_client_curl")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "ostream_span_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_ostream_span${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "otlp_http_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_otlp_http${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ else()
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ endif()
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST)
+ # Don't specify the DEPENDS unless we actually have dependencies, else
+ # Ninja/other build systems may consider this target to always be dirty
+ set(_OPENTELEMETRY_DEPENDENCIES)
+ add_custom_target(opentelemetry_dependencies)
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ set(_OPENTELEMETRY_DEPENDENCIES "opentelemetry_dependencies")
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ list(APPEND OPENTELEMETRY_PREFIX_PATH_LIST ${NLOHMANN_JSON_PREFIX})
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_OTLP=ON
+ -DWITH_OTLP_HTTP=ON -DWITH_OTLP_GRPC=OFF)
+
+ # OpenTelemetry with OTLP enabled requires Protobuf definitions from a
+ # submodule. This submodule path is hardcoded into their CMake definitions,
+ # and submodules are not included in their releases. Add a custom build step
+ # to download and extract the Protobufs.
+
+ # Adding such a step is rather complicated, so instead: create a separate
+ # ExternalProject that just fetches the Protobufs, then add a custom step
+ # to the main build to copy the Protobufs.
+ externalproject_add(opentelemetry_proto_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM}"
+ URL ${OPENTELEMETRY_PROTO_SOURCE_URL}
+ BUILD_COMMAND ""
+ CONFIGURE_COMMAND ""
+ INSTALL_COMMAND ""
+ EXCLUDE_FROM_ALL OFF)
+
+ add_dependencies(opentelemetry_dependencies nlohmann_json_ep opentelemetry_proto_ep)
+ if(gRPC_SOURCE STREQUAL "BUNDLED")
+ # TODO: opentelemetry-cpp::proto doesn't declare a dependency on gRPC, so
+ # even if we provide the location of gRPC, it'll fail to compile.
+ message(FATAL_ERROR "ARROW_WITH_OPENTELEMETRY cannot be configured with gRPC_SOURCE=BUNDLED. "
+ "See https://github.com/open-telemetry/opentelemetry-cpp/issues/1045"
+ )
+ endif()
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_API_ONLY=ON)
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR "|")
+ # JOIN is CMake >=3.12 only
+ string(REPLACE ";" "${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}"
+ OPENTELEMETRY_PREFIX_PATH "${OPENTELEMETRY_PREFIX_PATH_LIST}")
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ "-DCMAKE_PREFIX_PATH=${OPENTELEMETRY_PREFIX_PATH}")
+
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ # OpenTelemetry tries to determine the processor arch for vcpkg, which fails
+ # on s390x, even though it doesn't use vcpkg there. Tell it ARCH manually
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ endif()
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ externalproject_add_step(opentelemetry_ep download_proto
+ COMMAND ${CMAKE_COMMAND} -E copy_directory
+ $<TARGET_PROPERTY:opentelemetry_proto_ep,_EP_SOURCE_DIR>/opentelemetry
+ $<TARGET_PROPERTY:opentelemetry_ep,_EP_SOURCE_DIR>/third_party/opentelemetry-proto/opentelemetry
+ DEPENDEES download
+ DEPENDERS configure)
+ endif()
+
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+ set_target_properties(opentelemetry-cpp::http_client_curl
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::ext;CURL::libcurl")
+ set_target_properties(opentelemetry-cpp::proto
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "${ARROW_PROTOBUF_LIBPROTOBUF}")
+ set_target_properties(opentelemetry-cpp::otlp_recordable
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::trace;opentelemetry-cpp::resources;opentelemetry-cpp::proto"
+ )
+ set_target_properties(opentelemetry-cpp::otlp_http_exporter
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::otlp_recordable;opentelemetry-cpp::http_client_curl;nlohmann_json::nlohmann_json"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+if(ARROW_WITH_OPENTELEMETRY)
+ set(opentelemetry-cpp_SOURCE "AUTO")
+ if(ARROW_WITH_OPENTELEMETRY)
+ build_nlohmann_json_once()
+ find_curl()
Review comment:
Why should we put them here?
It seems that `build_opentelemetry()` should have them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #10260:
URL: https://github.com/apache/arrow/pull/10260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-983989733
Benchmark runs are scheduled for baseline = 200f1679f9837a83b105ba081f31ca47cbe22de1 and contender = bca06811e2da06a8c22de9ac0921329b29829521. bca06811e2da06a8c22de9ac0921329b29829521 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a8779635087d4dca91f81e1b36c31064...89f0b858dcbc41518455b77fb925f92a/)
[Failed :arrow_down:1.61% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/261584bbf92946cdad09914abd83954a...1d80c3c857cd4839809462f04cef946b/)
[Finished :arrow_down:0.44% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/84b1c177605b4549b54e8e8c4a09ed54...a841851bef404cdea0a686613f2f35c3/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10260: ARROW-12671: [C++][Python][FlightRPC] Simple OpenTelemetry integration
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-924987234
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r739465101
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
Ugh, enabling OTLP requires the bundled Protobuf definitions, which aren't in the release tarball (similar-ish issue before: https://github.com/open-telemetry/opentelemetry-cpp/issues/869)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-982977846
I think I've addressed all feedback here (thanks Kou for the detailed review!). Any other comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r751250174
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3865,6 +3905,221 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api ext sdk)
+ set(_OPENTELEMETRY_LIBS
+ common
+ http_client_curl
+ ostream_span_exporter
+ otlp_http_exporter
+ otlp_recordable
+ proto
+ resources
+ trace
+ version)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ if(_OPENTELEMETRY_LIB STREQUAL "http_client_curl")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "ostream_span_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_ostream_span${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ elseif(_OPENTELEMETRY_LIB STREQUAL "otlp_http_exporter")
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_exporter_otlp_http${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ else()
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ endif()
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST)
+ # Don't specify the DEPENDS unless we actually have dependencies, else
+ # Ninja/other build systems may consider this target to always be dirty
+ set(_OPENTELEMETRY_DEPENDENCIES)
+ add_custom_target(opentelemetry_dependencies)
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ set(_OPENTELEMETRY_DEPENDENCIES "opentelemetry_dependencies")
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ list(APPEND OPENTELEMETRY_PREFIX_PATH_LIST ${NLOHMANN_JSON_PREFIX})
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_OTLP=ON
+ -DWITH_OTLP_HTTP=ON -DWITH_OTLP_GRPC=OFF)
+
+ # OpenTelemetry with OTLP enabled requires Protobuf definitions from a
+ # submodule. This submodule path is hardcoded into their CMake definitions,
+ # and submodules are not included in their releases. Add a custom build step
+ # to download and extract the Protobufs.
+
+ # Adding such a step is rather complicated, so instead: create a separate
+ # ExternalProject that just fetches the Protobufs, then add a custom step
+ # to the main build to copy the Protobufs.
+ externalproject_add(opentelemetry_proto_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_PROTO_BUILD_SHA256_CHECKSUM}"
+ URL ${OPENTELEMETRY_PROTO_SOURCE_URL}
+ BUILD_COMMAND ""
+ CONFIGURE_COMMAND ""
+ INSTALL_COMMAND ""
+ EXCLUDE_FROM_ALL OFF)
+
+ add_dependencies(opentelemetry_dependencies nlohmann_json_ep opentelemetry_proto_ep)
+ if(gRPC_SOURCE STREQUAL "BUNDLED")
+ # TODO: opentelemetry-cpp::proto doesn't declare a dependency on gRPC, so
+ # even if we provide the location of gRPC, it'll fail to compile.
+ message(FATAL_ERROR "ARROW_WITH_OPENTELEMETRY cannot be configured with gRPC_SOURCE=BUNDLED. "
+ "See https://github.com/open-telemetry/opentelemetry-cpp/issues/1045"
+ )
+ endif()
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_API_ONLY=ON)
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+
+ set(OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR "|")
+ # JOIN is CMake >=3.12 only
+ string(REPLACE ";" "${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}"
+ OPENTELEMETRY_PREFIX_PATH "${OPENTELEMETRY_PREFIX_PATH_LIST}")
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ "-DCMAKE_PREFIX_PATH=${OPENTELEMETRY_PREFIX_PATH}")
+
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ # OpenTelemetry tries to determine the processor arch for vcpkg, which fails
+ # on s390x, even though it doesn't use vcpkg there. Tell it ARCH manually
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ LIST_SEPARATOR ${OPENTELEMETRY_PREFIX_PATH_LIST_SEP_CHAR}
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY}
+ DEPENDS ${_OPENTELEMETRY_DEPENDENCIES})
+ endif()
+
+ if(ARROW_WITH_OPENTELEMETRY)
+ externalproject_add_step(opentelemetry_ep download_proto
+ COMMAND ${CMAKE_COMMAND} -E copy_directory
+ $<TARGET_PROPERTY:opentelemetry_proto_ep,_EP_SOURCE_DIR>/opentelemetry
+ $<TARGET_PROPERTY:opentelemetry_ep,_EP_SOURCE_DIR>/third_party/opentelemetry-proto/opentelemetry
+ DEPENDEES download
+ DEPENDERS configure)
+ endif()
+
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ if(NOT ${CMAKE_VERSION} VERSION_LESS "3.15.0")
+ # Make sure all installed artifacts get cleaned up. Important for the RPM
+ # builds which fail if anything is left behind. Things in BUILD_BYPRODUCTS
+ # get cleaned automatically, but we can't list every header installed.
+ set_property(TARGET opentelemetry_ep
+ APPEND
+ PROPERTY ADDITIONAL_CLEAN_FILES "${OPENTELEMETRY_PREFIX}")
+ endif()
Review comment:
Ah, sorry, I left this behind. I've removed it.
It no longer runs any `opentelemetry_ep` steps in `ninja install`, though what I observed was just it generating stamp files, not actually running the install step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-984010527
> I'm curious: why are gRPC and protobuf required?
This PR configures the OTLP exporter so we can dump collected data somewhere where we can analyze it. The OTLP exporter talks to the OTLP collector over either HTTP or gRPC, using Protobuf payloads, so at least Protobuf and one of cURL and gRPC is required. We only enable the HTTP protocol, but the upstream CMake config currently doesn't differentiate between whether you have gRPC enabled or not when generating Protobuf stubs, so gRPC is unconditionally required. (See https://github.com/open-telemetry/opentelemetry-cpp/issues/1045.) I think this may have just been fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-983989733
Benchmark runs are scheduled for baseline = 200f1679f9837a83b105ba081f31ca47cbe22de1 and contender = bca06811e2da06a8c22de9ac0921329b29829521. bca06811e2da06a8c22de9ac0921329b29829521 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a8779635087d4dca91f81e1b36c31064...89f0b858dcbc41518455b77fb925f92a/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/261584bbf92946cdad09914abd83954a...1d80c3c857cd4839809462f04cef946b/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/84b1c177605b4549b54e8e8c4a09ed54...a841851bef404cdea0a686613f2f35c3/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cpcloud commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
cpcloud commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r736886675
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
+ public:
+ explicit OStreamJsonSpanExporter(std::ostream& sout = std::cerr) noexcept
+ : sout_(sout), shutdown_(false) {}
+ std::unique_ptr<sdktrace::Recordable> MakeRecordable() noexcept override {
+ return std::unique_ptr<sdktrace::Recordable>(new sdktrace::SpanData);
+ }
+ otel::sdk::common::ExportResult Export(
+ const nostd::span<std::unique_ptr<sdktrace::Recordable>>& spans) noexcept override {
+ if (shutdown_) return otel::sdk::common::ExportResult::kFailure;
+
+ for (auto& recordable : spans) {
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper> writer(sout_);
+ auto span = std::unique_ptr<sdktrace::SpanData>(
+ static_cast<sdktrace::SpanData*>(recordable.release()));
+ if (!span) continue;
+ char trace_id[32] = {0};
+ char span_id[16] = {0};
+ char parent_span_id[16] = {0};
+ span->GetTraceId().ToLowerBase16(trace_id);
+ span->GetSpanId().ToLowerBase16(span_id);
+ span->GetParentSpanId().ToLowerBase16(parent_span_id);
+
+ writer.StartObject();
+ writer.Key("name");
+ writer.String(span->GetName().data(), span->GetName().length());
+ writer.Key("trace_id");
+ writer.String(trace_id, 32);
+ writer.Key("span_id");
+ writer.String(span_id, 16);
+ writer.Key("parent_span_id");
+ writer.String(parent_span_id, 16);
+ writer.Key("start");
+ writer.Int64(span->GetStartTime().time_since_epoch().count());
+ writer.Key("duration");
+ writer.Int64(span->GetDuration().count());
+ writer.Key("description");
+ writer.String(span->GetDescription().data(), span->GetDescription().length());
+ writer.Key("kind");
+ writer.Int(static_cast<int>(span->GetSpanKind()));
+ writer.Key("status");
+ // TODO: this is expensive
+ writer.String(statuses_[static_cast<int>(span->GetStatus())]);
+ writer.Key("args");
+ writer.StartObject();
+ OwnedAttributeValueVisitor visitor(writer);
+ for (const auto& pair : span->GetAttributes()) {
+ writer.Key(pair.first.data(), pair.first.length());
+ nostd::visit(visitor, pair.second);
+ }
+ writer.EndObject();
+ writer.EndObject();
+ sout_.Put('\n');
+ }
+ sout_.Flush();
+ return otel::sdk::common::ExportResult::kSuccess;
+ }
+ bool Shutdown(std::chrono::microseconds) noexcept override {
+ shutdown_ = true;
+ return true;
+ }
+
+ private:
+ arrow::rapidjson::OStreamWrapper sout_;
+ bool shutdown_;
+ std::map<int, std::string> statuses_{{0, "Unset"}, {1, "Ok"}, {2, "Error"}};
+};
+#endif
+
+class ThreadIdSpanProcessor : public sdktrace::BatchSpanProcessor {
+ public:
+ using sdktrace::BatchSpanProcessor::BatchSpanProcessor;
+ void OnEnd(std::unique_ptr<sdktrace::Recordable>&& span) noexcept override {
+ std::stringstream thread_id;
+ thread_id << std::this_thread::get_id();
+ span->SetAttribute("thread_id", thread_id.str());
+ sdktrace::BatchSpanProcessor::OnEnd(std::move(span));
+ }
+};
+
+std::unique_ptr<sdktrace::SpanExporter> InitializeExporter() {
+ auto maybe_env_var = arrow::internal::GetEnvVar(kTracingBackendEnvVar);
+ if (maybe_env_var.ok()) {
+ auto env_var = maybe_env_var.ValueOrDie();
+ if (env_var == "json") {
+#ifdef ARROW_JSON
+ return std::unique_ptr<sdktrace::SpanExporter>(
+ new OStreamJsonSpanExporter(std::cerr));
+#else
+ ARROW_LOG(WARNING) << "Requested " << kTracingBackendEnvVar
+ << "=json but Arrow was built without ARROW_JSON";
+#endif
+ } else if (!env_var.empty()) {
+ ARROW_LOG(WARNING) << "Requested unknown backend " << kTracingBackendEnvVar << "="
+ << env_var;
+ }
+ }
+ return std::unique_ptr<sdktrace::SpanExporter>();
+}
+
+nostd::shared_ptr<sdktrace::TracerProvider> InitializeSdkTracerProvider() {
+ auto exporter = InitializeExporter();
+ if (exporter) {
+ sdktrace::BatchSpanProcessorOptions options;
+ options.max_queue_size = 16384;
+ options.schedule_delay_millis = std::chrono::milliseconds(500);
+ options.max_export_batch_size = 16384;
+ auto processor = std::unique_ptr<sdktrace::SpanProcessor>(
Review comment:
I guess this can't be `make_unique`?
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
Even if you don't go with the OTLP exporter (I assume the dependencies are annoying), you can use the `opentelemetry::exporter::trace::OStreamSpanExporter` which doesn't output JSON, but if you're not doing anything structured, it should suffice. The messages look like this:
```
{
name : library
trace_id : 2a93c0ffd8d36fce84c28b3d08f845c9
span_id : 109ea4dea0acdab5
tracestate :
parent_span_id: 0000000000000000
start : 1635282336724427669
duration : 75913
description :
span kind : Internal
status : Unset
attributes :
events :
links :
resources :
service.name: unknown_service
telemetry.sdk.version: 1.0.1
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: cpp
instr-lib : foo_library
}
```
For any structure, the collector can be used.
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
+ public:
+ explicit OStreamJsonSpanExporter(std::ostream& sout = std::cerr) noexcept
+ : sout_(sout), shutdown_(false) {}
+ std::unique_ptr<sdktrace::Recordable> MakeRecordable() noexcept override {
+ return std::unique_ptr<sdktrace::Recordable>(new sdktrace::SpanData);
+ }
+ otel::sdk::common::ExportResult Export(
+ const nostd::span<std::unique_ptr<sdktrace::Recordable>>& spans) noexcept override {
+ if (shutdown_) return otel::sdk::common::ExportResult::kFailure;
+
+ for (auto& recordable : spans) {
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper> writer(sout_);
+ auto span = std::unique_ptr<sdktrace::SpanData>(
+ static_cast<sdktrace::SpanData*>(recordable.release()));
+ if (!span) continue;
+ char trace_id[32] = {0};
+ char span_id[16] = {0};
+ char parent_span_id[16] = {0};
+ span->GetTraceId().ToLowerBase16(trace_id);
+ span->GetSpanId().ToLowerBase16(span_id);
+ span->GetParentSpanId().ToLowerBase16(parent_span_id);
+
+ writer.StartObject();
+ writer.Key("name");
+ writer.String(span->GetName().data(), span->GetName().length());
+ writer.Key("trace_id");
+ writer.String(trace_id, 32);
+ writer.Key("span_id");
+ writer.String(span_id, 16);
+ writer.Key("parent_span_id");
+ writer.String(parent_span_id, 16);
+ writer.Key("start");
+ writer.Int64(span->GetStartTime().time_since_epoch().count());
+ writer.Key("duration");
+ writer.Int64(span->GetDuration().count());
+ writer.Key("description");
+ writer.String(span->GetDescription().data(), span->GetDescription().length());
+ writer.Key("kind");
+ writer.Int(static_cast<int>(span->GetSpanKind()));
+ writer.Key("status");
+ // TODO: this is expensive
+ writer.String(statuses_[static_cast<int>(span->GetStatus())]);
+ writer.Key("args");
+ writer.StartObject();
+ OwnedAttributeValueVisitor visitor(writer);
+ for (const auto& pair : span->GetAttributes()) {
+ writer.Key(pair.first.data(), pair.first.length());
+ nostd::visit(visitor, pair.second);
+ }
+ writer.EndObject();
+ writer.EndObject();
+ sout_.Put('\n');
+ }
+ sout_.Flush();
+ return otel::sdk::common::ExportResult::kSuccess;
+ }
+ bool Shutdown(std::chrono::microseconds) noexcept override {
+ shutdown_ = true;
+ return true;
+ }
+
+ private:
+ arrow::rapidjson::OStreamWrapper sout_;
+ bool shutdown_;
+ std::map<int, std::string> statuses_{{0, "Unset"}, {1, "Ok"}, {2, "Error"}};
+};
+#endif
+
+class ThreadIdSpanProcessor : public sdktrace::BatchSpanProcessor {
+ public:
+ using sdktrace::BatchSpanProcessor::BatchSpanProcessor;
+ void OnEnd(std::unique_ptr<sdktrace::Recordable>&& span) noexcept override {
+ std::stringstream thread_id;
+ thread_id << std::this_thread::get_id();
+ span->SetAttribute("thread_id", thread_id.str());
+ sdktrace::BatchSpanProcessor::OnEnd(std::move(span));
+ }
+};
+
+std::unique_ptr<sdktrace::SpanExporter> InitializeExporter() {
+ auto maybe_env_var = arrow::internal::GetEnvVar(kTracingBackendEnvVar);
+ if (maybe_env_var.ok()) {
+ auto env_var = maybe_env_var.ValueOrDie();
+ if (env_var == "json") {
+#ifdef ARROW_JSON
+ return std::unique_ptr<sdktrace::SpanExporter>(
+ new OStreamJsonSpanExporter(std::cerr));
+#else
+ ARROW_LOG(WARNING) << "Requested " << kTracingBackendEnvVar
+ << "=json but Arrow was built without ARROW_JSON";
+#endif
+ } else if (!env_var.empty()) {
+ ARROW_LOG(WARNING) << "Requested unknown backend " << kTracingBackendEnvVar << "="
+ << env_var;
+ }
+ }
+ return std::unique_ptr<sdktrace::SpanExporter>();
+}
+
+nostd::shared_ptr<sdktrace::TracerProvider> InitializeSdkTracerProvider() {
+ auto exporter = InitializeExporter();
+ if (exporter) {
+ sdktrace::BatchSpanProcessorOptions options;
+ options.max_queue_size = 16384;
+ options.schedule_delay_millis = std::chrono::milliseconds(500);
+ options.max_export_batch_size = 16384;
+ auto processor = std::unique_ptr<sdktrace::SpanProcessor>(
+ new ThreadIdSpanProcessor(std::move(exporter), options));
+ return nostd::shared_ptr<sdktrace::TracerProvider>(
Review comment:
Similarly, guessing this can't be `make_shared`?
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
I'm able to run these examples (https://github.com/open-telemetry/opentelemetry-cpp/tree/main/examples/otlp) with a tiny collector setup producing JSON to stdout. I don't think this exporter is necessary.
Additionally, I think we'd lose a lot of the benefits of opentelemetry by outputting JSON that isn't compatible with any otel consumer. If you run the examples you'll see that the JSON differs what's being written here in a number of meaningful ways.
I think it's enough to add the initialization (sans this exporter) and the async generator wrapper.
To reproduce the examples:
1. Build the library from source (the examples should build by default)
2. change `examples/otlp/opentelemetry-collector-config/config.dev.yaml` to this:
```yaml
exporters:
file:
path: /dev/stdout
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: "0.0.0.0:4318"
cors_allowed_origins:
- '*'
service:
pipelines:
traces:
receivers:
- otlp
exporters:
- file
```
3. Start the collector container from the `opentelemetry-cpp` clone:
```
docker run --rm -it -p 4317:4317 -p 4318:4318 -v $(pwd)/examples/otlp:/cfg otel/opentelemetry-collector:latest --config=/cfg/opentelemetry-collector-config/config.dev.yaml
```
If you now run `build/examples/otlp/example_otlp_grpc` or `build/examples/otlp/example_otlp_http` you should see a line of JSON coming out from the place you launch the container for each invocation.
The example can be simplified further by removing one of the `grpc` or `http` receivers.
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
+ public:
+ explicit OStreamJsonSpanExporter(std::ostream& sout = std::cerr) noexcept
+ : sout_(sout), shutdown_(false) {}
+ std::unique_ptr<sdktrace::Recordable> MakeRecordable() noexcept override {
+ return std::unique_ptr<sdktrace::Recordable>(new sdktrace::SpanData);
+ }
+ otel::sdk::common::ExportResult Export(
+ const nostd::span<std::unique_ptr<sdktrace::Recordable>>& spans) noexcept override {
+ if (shutdown_) return otel::sdk::common::ExportResult::kFailure;
+
+ for (auto& recordable : spans) {
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper> writer(sout_);
+ auto span = std::unique_ptr<sdktrace::SpanData>(
+ static_cast<sdktrace::SpanData*>(recordable.release()));
+ if (!span) continue;
+ char trace_id[32] = {0};
+ char span_id[16] = {0};
+ char parent_span_id[16] = {0};
+ span->GetTraceId().ToLowerBase16(trace_id);
+ span->GetSpanId().ToLowerBase16(span_id);
+ span->GetParentSpanId().ToLowerBase16(parent_span_id);
+
+ writer.StartObject();
+ writer.Key("name");
+ writer.String(span->GetName().data(), span->GetName().length());
+ writer.Key("trace_id");
+ writer.String(trace_id, 32);
+ writer.Key("span_id");
+ writer.String(span_id, 16);
+ writer.Key("parent_span_id");
+ writer.String(parent_span_id, 16);
+ writer.Key("start");
+ writer.Int64(span->GetStartTime().time_since_epoch().count());
+ writer.Key("duration");
+ writer.Int64(span->GetDuration().count());
+ writer.Key("description");
+ writer.String(span->GetDescription().data(), span->GetDescription().length());
+ writer.Key("kind");
+ writer.Int(static_cast<int>(span->GetSpanKind()));
+ writer.Key("status");
+ // TODO: this is expensive
+ writer.String(statuses_[static_cast<int>(span->GetStatus())]);
+ writer.Key("args");
+ writer.StartObject();
+ OwnedAttributeValueVisitor visitor(writer);
+ for (const auto& pair : span->GetAttributes()) {
+ writer.Key(pair.first.data(), pair.first.length());
+ nostd::visit(visitor, pair.second);
+ }
+ writer.EndObject();
+ writer.EndObject();
+ sout_.Put('\n');
+ }
+ sout_.Flush();
+ return otel::sdk::common::ExportResult::kSuccess;
+ }
+ bool Shutdown(std::chrono::microseconds) noexcept override {
+ shutdown_ = true;
+ return true;
+ }
+
+ private:
+ arrow::rapidjson::OStreamWrapper sout_;
+ bool shutdown_;
+ std::map<int, std::string> statuses_{{0, "Unset"}, {1, "Ok"}, {2, "Error"}};
+};
+#endif
+
+class ThreadIdSpanProcessor : public sdktrace::BatchSpanProcessor {
Review comment:
Is this the thread of the current span, or the thread of the thing doing the processing (these could in theory be separate threads)
##########
File path: cpp/src/arrow/util/tracing_internal.cc
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/tracing_internal.h"
+
+#include <iostream>
+#include <sstream>
+#include <thread>
+
+#ifdef _MSC_VER
+#pragma warning(push)
+#pragma warning(disable : 4522)
+#endif
+#ifdef ARROW_WITH_OPENTELEMETRY
+#include <opentelemetry/sdk/trace/batch_span_processor.h>
+#include <opentelemetry/sdk/trace/recordable.h>
+#include <opentelemetry/sdk/trace/span_data.h>
+#include <opentelemetry/sdk/trace/tracer_provider.h>
+#include <opentelemetry/trace/noop.h>
+#include <opentelemetry/trace/provider.h>
+#endif
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+#include "arrow/util/config.h"
+#include "arrow/util/io_util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/make_unique.h"
+#ifdef ARROW_JSON
+#include "arrow/json/rapidjson_defs.h"
+#include "rapidjson/ostreamwrapper.h"
+#include "rapidjson/writer.h"
+#endif
+
+namespace arrow {
+namespace internal {
+namespace tracing {
+
+namespace nostd = opentelemetry::nostd;
+namespace otel = opentelemetry;
+
+constexpr char kTracingBackendEnvVar[] = "ARROW_TRACING_BACKEND";
+
+namespace {
+
+#ifdef ARROW_WITH_OPENTELEMETRY
+namespace sdktrace = opentelemetry::sdk::trace;
+#ifdef ARROW_JSON
+struct OwnedAttributeValueVisitor {
+ OwnedAttributeValueVisitor(
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer_)
+ : writer(writer_) {}
+
+ void operator()(const std::string& arg) { writer.String(arg); }
+
+ void operator()(const int32_t& arg) { writer.Int(arg); }
+
+ void operator()(const uint32_t& arg) { writer.Uint(arg); }
+
+ void operator()(const int64_t& arg) { writer.Int64(arg); }
+
+ void operator()(const uint64_t& arg) { writer.Uint64(arg); }
+
+ template <typename T>
+ void operator()(T&& arg) {
+ writer.Null();
+ }
+
+ arrow::rapidjson::Writer<arrow::rapidjson::OStreamWrapper>& writer;
+};
+
+/// Export spans as newline-delimited JSON.
+class OStreamJsonSpanExporter : public sdktrace::SpanExporter {
Review comment:
Is this actually necessary? Typically a collector handles turning the otel data into `$MY_FAVORITE_FORMAT`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-956342173
I don't have that right now, unfortunately - the other issue is that if we had such a C++ application then this would be moot (you would just link and configure OTel yourself), here I'm trying to just target the case of the Python/R bindings and for local development.
I could strip all this out and maybe leave only the OStream exporter for local development, but that doesn't really help (say) people trying to analyze the R TPC-H benchmarks since they won't be able to get the data in any usable form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#issuecomment-967350296
@github-actions crossbow submit -g nightly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #10260: ARROW-12671: [C++] Add OpenTelemetry to ThirdpartyToolchain
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #10260:
URL: https://github.com/apache/arrow/pull/10260#discussion_r749684986
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -162,6 +163,8 @@ macro(build_dependency DEPENDENCY_NAME)
build_lz4()
elseif("${DEPENDENCY_NAME}" STREQUAL "ORC")
build_orc()
+ elseif("${DEPENDENCY_NAME}" STREQUAL "opentelemetry-cpp")
+ build_opentelemetry()
Review comment:
```suggestion
elseif("${DEPENDENCY_NAME}" STREQUAL "opentelemetry-cpp")
build_opentelemetry()
elseif("${DEPENDENCY_NAME}" STREQUAL "ORC")
build_orc()
```
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3661,17 +3711,7 @@ macro(build_google_cloud_cpp_storage)
add_dependencies(google_cloud_cpp_dependencies nlohmann_json_ep)
# Typically the steps to build the AWKSSDK provide `CURL::libcurl`, but if that is
# disabled we need to provide our own.
- if(NOT TARGET CURL::libcurl)
- find_package(CURL REQUIRED)
- if(NOT TARGET CURL::libcurl)
- # For CMake 3.11 or older
- add_library(CURL::libcurl UNKNOWN IMPORTED)
- set_target_properties(CURL::libcurl
- PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
- "${CURL_INCLUDE_DIRS}" IMPORTED_LOCATION
- "${CURL_LIBRARIES}")
- endif()
- endif()
+ find_curl()
Review comment:
There is one more `find_package(CURL)` in `build_google_cloud_cpp_storage`.
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -3839,6 +3851,115 @@ if(ARROW_ORC)
message(STATUS "Found ORC headers: ${ORC_INCLUDE_DIR}")
endif()
+# ----------------------------------------------------------------------
+# OpenTelemetry C++
+
+macro(build_opentelemetry)
+ message("Building OpenTelemetry from source")
+ set(OPENTELEMETRY_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/opentelemetry_ep-install")
+ set(OPENTELEMETRY_INCLUDE_DIR "${OPENTELEMETRY_PREFIX}/include")
+ set(OPENTELEMETRY_STATIC_LIB
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ set(_OPENTELEMETRY_APIS api sdk)
+ set(_OPENTELEMETRY_LIBS common resources trace)
+ set(OPENTELEMETRY_BUILD_BYPRODUCTS)
+ set(OPENTELEMETRY_LIBRARIES)
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_APIS})
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} INTERFACE IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES INTERFACE_INCLUDE_DIRECTORIES
+ "${OPENTELEMETRY_INCLUDE_DIR}")
+ endforeach()
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ set(_OPENTELEMETRY_STATIC_LIBRARY
+ "${OPENTELEMETRY_PREFIX}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}opentelemetry_${_OPENTELEMETRY_LIB}${CMAKE_STATIC_LIBRARY_SUFFIX}"
+ )
+ add_library(opentelemetry-cpp::${_OPENTELEMETRY_LIB} STATIC IMPORTED)
+ set_target_properties(opentelemetry-cpp::${_OPENTELEMETRY_LIB}
+ PROPERTIES IMPORTED_LOCATION ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_BUILD_BYPRODUCTS ${_OPENTELEMETRY_STATIC_LIBRARY})
+ list(APPEND OPENTELEMETRY_LIBRARIES opentelemetry-cpp::${_OPENTELEMETRY_LIB})
+ endforeach()
+
+ set(OPENTELEMETRY_CMAKE_ARGS
+ ${EP_COMMON_TOOLCHAIN}
+ "-DCMAKE_INSTALL_PREFIX=${OPENTELEMETRY_PREFIX}"
+ "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+ -DCMAKE_INSTALL_LIBDIR=lib
+ "-DCMAKE_CXX_FLAGS=${EP_CXX_FLAGS}"
+ -DBUILD_TESTING=OFF
+ -DWITH_EXAMPLES=OFF)
+ if(ARROW_WITH_OPENTELEMETRY)
+ list(APPEND ARROW_BUNDLED_STATIC_LIBS ${OPENTELEMETRY_LIBRARIES})
+ else()
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} "-DWITH_API_ONLY=ON")
+ if(WIN32)
+ # WITH_ETW does not respect WITH_API_ONLY
+ set(OPENTELEMETRY_CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS} -DWITH_ETW=OFF)
+ endif()
+ endif()
+ if(CMAKE_SYSTEM_PROCESSOR STREQUAL "s390x")
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ # OpenTelemetry tries to determine the processor arch for vcpkg,
+ # which fails on s390x, even though it doesn't use vcpkg there
+ CONFIGURE_COMMAND ${CMAKE_COMMAND} -E env ARCH=s390x
+ ${CMAKE_COMMAND} -G ${CMAKE_GENERATOR}
+ "<SOURCE_DIR><SOURCE_SUBDIR>"
+ ${OPENTELEMETRY_CMAKE_ARGS}
+ BUILD_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target all
+ INSTALL_COMMAND ${CMAKE_COMMAND} --build "<BINARY_DIR>" --target
+ install
+ # CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ else()
+ externalproject_add(opentelemetry_ep
+ ${EP_LOG_OPTIONS}
+ URL_HASH "SHA256=${ARROW_OPENTELEMETRY_BUILD_SHA256_CHECKSUM}"
+ CMAKE_ARGS ${OPENTELEMETRY_CMAKE_ARGS}
+ URL ${OPENTELEMETRY_SOURCE_URL}
+ BUILD_BYPRODUCTS ${OPENTELEMETRY_BUILD_BYPRODUCTS}
+ EXCLUDE_FROM_ALL NOT
+ ${ARROW_WITH_OPENTELEMETRY})
+ endif()
+ add_dependencies(toolchain opentelemetry_ep)
+ add_dependencies(toolchain-tests opentelemetry_ep)
+
+ set(OPENTELEMETRY_VENDORED 1)
+
+ set_target_properties(opentelemetry-cpp::common
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::api;opentelemetry-cpp::sdk;Threads::Threads"
+ )
+ set_target_properties(opentelemetry-cpp::resources
+ PROPERTIES INTERFACE_LINK_LIBRARIES "opentelemetry-cpp::common")
+ set_target_properties(opentelemetry-cpp::trace
+ PROPERTIES INTERFACE_LINK_LIBRARIES
+ "opentelemetry-cpp::common;opentelemetry-cpp::resources"
+ )
+
+ foreach(_OPENTELEMETRY_LIB ${_OPENTELEMETRY_LIBS})
+ add_dependencies(opentelemetry-cpp::${_OPENTELEMETRY_LIB} opentelemetry_ep)
+ endforeach()
+endmacro()
+
+# For now OpenTelemetry is always bundled from upstream
+if(1)
Review comment:
I'm not sure why do we need OpenTelemetry even when we don't enable Flight...
##########
File path: cpp/src/arrow/CMakeLists.txt
##########
@@ -575,6 +576,12 @@ if(ARROW_WITH_BACKTRACE)
endforeach()
endif()
+if(ARROW_WITH_OPENTELEMETRY)
+ foreach(LIB_TARGET ${ARROW_LIBRARIES})
+ target_compile_definitions(${LIB_TARGET} PRIVATE ARROW_WITH_OPENTELEMETRY)
Review comment:
Could you use `cpp/src/arrow/util/config.h.cmake` instead?
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -225,7 +228,7 @@ macro(resolve_dependency DEPENDENCY_NAME)
list(APPEND FIND_PACKAGE_ARGUMENTS CONFIG)
endif()
if(${DEPENDENCY_NAME}_SOURCE STREQUAL "AUTO")
- find_package(${FIND_PACKAGE_ARGUMENTS})
+ find_package(${FIND_PACKAGE_ARGUMENTS} QUIET)
Review comment:
Why do you want to add `QUIET` here?
I think that `find_package()` output is useful for debugging, user support and so on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org