You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by br...@apache.org on 2024/11/11 21:46:10 UTC
(arrow) branch maint-18.1.0 created (now 2354c2f386)
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a change to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
at 2354c2f386 GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN (#44685)
This branch includes the following new commits:
new 47ba57030c GH-44353: [Java] Implement `map()` for `UnionMapWriter` (#44390)
new f42c5b0c24 GH-44361: [C#][Integration] Include .NET in Flight integration tests (#44377)
new a803dd9df6 GH-44360: [C#] Fix Flight DoExchange incompatibility with C++ implementation (#44424)
new c13bed995f GH-44389: [Java][Integration][Release] Use Python 3.12 for verify-rc-source-integration-linux-conda-latest-amd64 (#44395)
new c4cd5a263f GH-44453: [Release] Add shebang to `07-matlab-upload.sh` (#44454)
new 6040c07cea GH-44451: [Release] Retry on HTTP error in Java upload (#44452)
new 587af2e676 GH-44459: [Release][Packaging] Do not check wheel content when verifying binary wheels (#44460)
new bc8251f1cc GH-44449: [Release] Retry on HTTP error in binary upload (#44450)
new e56e0a48e7 GH-34535: [C++] Move ChunkResolver to the public API (#44357)
new 9e7feb94b5 GH-44506: [Docs][C++] Fix documentation directive for ChunkLocation (#44505)
new 925c96528d GH-44448: [C++] Add support for overwriting grpc_cpp_plugin path for cross-compiling (#44507)
new 7e26edf033 GH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on verify-release-candidate-wheels.bat (#44462)
new abd021c855 GH-44606: [C++] Add find module for abseil that handles missing version (#44613)
new d20173f488 GH-44674: [R] Fix R CMD check failure with dev testthat (#44675)
new 2354c2f386 GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN (#44685)
The 15 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(arrow) 13/15: GH-44606: [C++] Add find module for abseil that handles missing version (#44613)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit abd021c85542b77ad61a033c6dbaaac3603f6eef
Author: Jacob Wujciak-Jens <ja...@wujciak.de>
AuthorDate: Fri Nov 8 02:25:29 2024 +0100
GH-44606: [C++] Add find module for abseil that handles missing version (#44613)
### Rationale for this change
Abseil's recommended usage is to build from trunk, this version does not provide a package version in CMake which breaks `resolve_dependency`.
### What changes are included in this PR?
Add `FindabslAlt.cmake` that adds a big version if it is missing to compare against our minimal version requirements.
### Are these changes tested?
CI
### Are there any user-facing changes?
Abseil build from head should now work without workaround.
* GitHub Issue: #44606
Lead-authored-by: Jacob Wujciak-Jens <ja...@wujciak.de>
Co-authored-by: Sutou Kouhei <ko...@cozmixng.org>
Signed-off-by: Jacob Wujciak-Jens <ja...@wujciak.de>
---
cpp/cmake_modules/FindabslAlt.cmake | 46 +++++++++++++++++++++++++++++
cpp/cmake_modules/ThirdpartyToolchain.cmake | 2 +-
2 files changed, 47 insertions(+), 1 deletion(-)
diff --git a/cpp/cmake_modules/FindabslAlt.cmake b/cpp/cmake_modules/FindabslAlt.cmake
new file mode 100644
index 0000000000..a7ebe63f5e
--- /dev/null
+++ b/cpp/cmake_modules/FindabslAlt.cmake
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if(abslAlt_FOUND)
+ return()
+endif()
+
+set(find_package_args)
+
+if(abslAlt_FIND_QUIETLY)
+ list(APPEND find_package_args QUIET)
+endif()
+if(abslAlt_FIND_REQUIRED)
+ list(APPEND find_package_args REQUIRED)
+endif()
+
+find_package(absl ${find_package_args})
+
+if(NOT DEFINED absl_VERSION)
+ # Abseil does not define a version when build 'live at head'.
+ # As this is their recommended path we need to define a large version to pass version checks.
+ # CMake removes the '_head' suffix for version comparison but it will show up in the logs
+ # and matches the abseil-cpp.pc version of 'head'
+ set(absl_VERSION 99999999_head)
+endif()
+
+set(abslAlt_VERSION ${absl_VERSION})
+
+find_package_handle_standard_args(
+ abslAlt
+ REQUIRED_VARS absl_FOUND
+ VERSION_VAR abslAlt_VERSION)
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 0b215b5b25..e07e13cab5 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4169,7 +4169,7 @@ if(ARROW_WITH_GOOGLE_CLOUD_CPP OR ARROW_WITH_GRPC)
ARROW_PC_PACKAGE_NAME
${ARROW_ABSL_PC_PACKAGE_NAME}
HAVE_ALT
- FALSE
+ TRUE
FORCE_ANY_NEWER_VERSION
TRUE
REQUIRED_VERSION
(arrow) 01/15: GH-44353: [Java] Implement `map()` for `UnionMapWriter` (#44390)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 47ba57030cf05f77791a03b3d00ba48cae5bb2c2
Author: 0xderek <xf...@gmail.com>
AuthorDate: Tue Oct 15 08:41:45 2024 +0800
GH-44353: [Java] Implement `map()` for `UnionMapWriter` (#44390)
### Rationale for this change
See #44353
`UnionMapWriter` does not override `map()` and falls back to implementation in `UnionListWriter` which causes exception when used.
### What changes are included in this PR?
Implement `map()` for `UnionMapWriter`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Changed behavior of the related function.
* GitHub Issue: #44353
Authored-by: 0xderek <al...@gmail.com>
Signed-off-by: David Li <li...@gmail.com>
---
.../src/main/codegen/templates/UnionMapWriter.java | 12 ++++++
.../org/apache/arrow/vector/TestMapVector.java | 49 +++++++++++++++++++---
2 files changed, 55 insertions(+), 6 deletions(-)
diff --git a/java/vector/src/main/codegen/templates/UnionMapWriter.java b/java/vector/src/main/codegen/templates/UnionMapWriter.java
index 606f880377..90b55cb65e 100644
--- a/java/vector/src/main/codegen/templates/UnionMapWriter.java
+++ b/java/vector/src/main/codegen/templates/UnionMapWriter.java
@@ -219,4 +219,16 @@ public class UnionMapWriter extends UnionListWriter {
return super.map();
}
}
+
+ @Override
+ public MapWriter map() {
+ switch (mode) {
+ case KEY:
+ return entryWriter.map(MapVector.KEY_NAME);
+ case VALUE:
+ return entryWriter.map(MapVector.VALUE_NAME);
+ default:
+ return super.map();
+ }
+ }
}
diff --git a/java/vector/src/test/java/org/apache/arrow/vector/TestMapVector.java b/java/vector/src/test/java/org/apache/arrow/vector/TestMapVector.java
index 213ffced27..a4197c50b5 100644
--- a/java/vector/src/test/java/org/apache/arrow/vector/TestMapVector.java
+++ b/java/vector/src/test/java/org/apache/arrow/vector/TestMapVector.java
@@ -640,11 +640,12 @@ public class TestMapVector {
MapWriter valueWriter;
// we are essentially writing Map<Long, Map<Long, Long>>
- // populate map vector with the following three records
+ // populate map vector with the following four records
// [
// null,
// [1:[50: 100, 200:400], 2:[75: 175, 150: 250]],
- // [3:[10: 20], 4:[15: 20], 5:[25: 30, 35: null]]
+ // [3:[10: 20], 4:[15: 20], 5:[25: 30, 35: null]],
+ // [8:[15: 30, 10: 20]]
// ]
/* write null at index 0 */
@@ -706,11 +707,26 @@ public class TestMapVector {
mapWriter.endMap();
- assertEquals(2, mapVector.getLastSet());
+ /* write one or more maps at index 3 */
+ mapWriter.setPosition(3);
+ mapWriter.startMap();
+
+ mapWriter.startEntry();
+ mapWriter.key().bigInt().writeBigInt(8);
+ valueWriter = mapWriter.value().map();
+ valueWriter.startMap();
+ writeEntry(valueWriter, 15, 30L);
+ writeEntry(valueWriter, 10, 20L);
+ valueWriter.endMap();
+ mapWriter.endEntry();
+
+ mapWriter.endMap();
+
+ assertEquals(3, mapVector.getLastSet());
- mapWriter.setValueCount(3);
+ mapWriter.setValueCount(4);
- assertEquals(3, mapVector.getValueCount());
+ assertEquals(4, mapVector.getValueCount());
// Get mapVector element at index 0
Object result = mapVector.getObject(0);
@@ -784,19 +800,40 @@ public class TestMapVector {
assertEquals(35L, getResultKey(innerMap));
assertNull(innerMap.get(MapVector.VALUE_NAME));
+ // Get mapVector element at index 3
+ result = mapVector.getObject(3);
+ resultSet = (ArrayList<?>) result;
+
+ // only 1 map entry at index 3
+ assertEquals(1, resultSet.size());
+
+ resultStruct = (Map<?, ?>) resultSet.get(0);
+ assertEquals(8L, getResultKey(resultStruct));
+ list = (ArrayList<Map<?, ?>>) getResultValue(resultStruct);
+ assertEquals(2, list.size()); // value is a list of 2 maps
+ innerMap = list.get(0);
+ assertEquals(15L, getResultKey(innerMap));
+ assertEquals(30L, getResultValue(innerMap));
+ innerMap = list.get(1);
+ assertEquals(10L, getResultKey(innerMap));
+ assertEquals(20L, getResultValue(innerMap));
+
/* check underlying bitVector */
assertTrue(mapVector.isNull(0));
assertFalse(mapVector.isNull(1));
assertFalse(mapVector.isNull(2));
+ assertFalse(mapVector.isNull(3));
/* check underlying offsets */
final ArrowBuf offsetBuffer = mapVector.getOffsetBuffer();
- /* mapVector has 0 entries at index 0, 2 entries at index 1, and 3 entries at index 2 */
+ // mapVector has 0 entries at index 0, 2 entries at index 1, 3 entries at index 2,
+ // and 1 entry at index 3
assertEquals(0, offsetBuffer.getInt(0 * MapVector.OFFSET_WIDTH));
assertEquals(0, offsetBuffer.getInt(1 * MapVector.OFFSET_WIDTH));
assertEquals(2, offsetBuffer.getInt(2 * MapVector.OFFSET_WIDTH));
assertEquals(5, offsetBuffer.getInt(3 * MapVector.OFFSET_WIDTH));
+ assertEquals(6, offsetBuffer.getInt(4 * MapVector.OFFSET_WIDTH));
}
}
(arrow) 08/15: GH-44449: [Release] Retry on HTTP error in binary upload (#44450)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit bc8251f1cc6264404f2ec86305c4301bde307aef
Author: Sutou Kouhei <ko...@clear-code.com>
AuthorDate: Fri Oct 18 04:59:55 2024 +0900
GH-44449: [Release] Retry on HTTP error in binary upload (#44450)
### Rationale for this change
Artifactory sometimes reports the following 503 HTTP error:
```text
BinaryTask::ArtifactoryClient::Error: failed to request: https://apache.jfrog.io/artifactory/arrow/debian-rc/pool/bookworm/main/a/apache-arrow/libparquet-glib1800_18.0.0-1_arm64.deb: PUT: 503
{
"errors" : [ {
"status" : 503,
"message" : "503 : Failed during addStream to bucket"
} ]
}
```
It seems that this is a temporary error. So we can retry on this type error for stable upload.
### What changes are included in this PR?
Retry on HTTP error.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #44449
Authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
---
dev/release/binary-task.rb | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/dev/release/binary-task.rb b/dev/release/binary-task.rb
index b06b1c46b8..4387641741 100644
--- a/dev/release/binary-task.rb
+++ b/dev/release/binary-task.rb
@@ -532,7 +532,8 @@ class BinaryTask
OpenSSL::OpenSSLError,
SocketError,
SystemCallError,
- Timeout::Error => error
+ Timeout::Error,
+ Error => error
n_retries += 1
if n_retries <= max_n_retries
$stderr.puts
(arrow) 11/15: GH-44448: [C++] Add support for overwriting grpc_cpp_plugin path for cross-compiling (#44507)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 925c96528dfc9fc76f65b6a8f27662ba27b7f72c
Author: Sutou Kouhei <ko...@clear-code.com>
AuthorDate: Wed Oct 23 14:04:57 2024 +0900
GH-44448: [C++] Add support for overwriting grpc_cpp_plugin path for cross-compiling (#44507)
### Rationale for this change
We can't use `find_package(gRPC)` and `gRPC::grpc_cpp_plugin` for cross-compiling because it's for host. We need `grpc_cpp_plugin` for target in cross-compiling.
### What changes are included in this PR?
Add `ARROW_GRPC_CPP_PLUGIN` CMake option that overwrites `gRPC::grpc_cpp_plugin` path found by `find_package(gRPC)`.
### Are these changes tested?
Yes.
https://github.com/conda-forge/arrow-cpp-feedstock/pull/1432
### Are there any user-facing changes?
Yes.
* GitHub Issue: #44448
Authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
---
cpp/cmake_modules/DefineOptions.cmake | 5 +++++
cpp/cmake_modules/ThirdpartyToolchain.cmake | 8 ++++++++
2 files changed, 13 insertions(+)
diff --git a/cpp/cmake_modules/DefineOptions.cmake b/cpp/cmake_modules/DefineOptions.cmake
index d823444cf7..a7bf9e59f8 100644
--- a/cpp/cmake_modules/DefineOptions.cmake
+++ b/cpp/cmake_modules/DefineOptions.cmake
@@ -640,6 +640,11 @@ Always OFF if building binaries" OFF)
"Compiler flags to append when pre-compiling Gandiva operations"
"")
+ #----------------------------------------------------------------------
+ set_option_category("Cross compiling")
+
+ define_option_string(ARROW_GRPC_CPP_PLUGIN "grpc_cpp_plugin path to be used" "")
+
#----------------------------------------------------------------------
set_option_category("Advanced developer")
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index db151b4e0f..0b215b5b25 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -4223,6 +4223,14 @@ if(ARROW_WITH_GRPC)
target_link_libraries(gRPC::grpc++ INTERFACE gRPC::grpc_asan_suppressed)
endif()
endif()
+
+ if(ARROW_GRPC_CPP_PLUGIN)
+ if(NOT TARGET gRPC::grpc_cpp_plugin)
+ add_executable(gRPC::grpc_cpp_plugin IMPORTED)
+ endif()
+ set_target_properties(gRPC::grpc_cpp_plugin PROPERTIES IMPORTED_LOCATION
+ ${ARROW_GRPC_CPP_PLUGIN})
+ endif()
endif()
# ----------------------------------------------------------------------
(arrow) 09/15: GH-34535: [C++] Move ChunkResolver to the public API (#44357)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit e56e0a48e71b798cfd36078b4c2f1a54b852153a
Author: Anja Kefala <an...@gmail.com>
AuthorDate: Mon Oct 21 15:46:48 2024 -0700
GH-34535: [C++] Move ChunkResolver to the public API (#44357)
### Rationale for this change
Adopting https://github.com/apache/arrow/pull/40226.
The creation and return of a shared_ptr does result in some performance overhead, that makes a difference for a performance-sensitive application.
If someone could use ChunkResolver to learn the indices, they could then instead access the data directly.
### What changes are included in this PR?
- [X] Updates to documentation (thanks to @ SChakravorti21 )
- [X] Moving `ChunkResolver` to public API, and updating all references to it in the code
### Are these changes tested?
There seemed to be comprehensive tests already: https://github.com/apache/arrow/blob/main/cpp/src/arrow/chunked_array_test.cc#L324 If an edgecase is missing, I'd be happy to add it.
### Are there any user-facing changes?
`ChunkResolver` and `TypedChunkLocation` are now in the public API.
* GitHub Issue: #34535
Lead-authored-by: Anja Kefala <an...@gmail.com>
Co-authored-by: anjakefala <an...@gmail.com>
Co-authored-by: Bryce Mecum <pe...@gmail.com>
Co-authored-by: SChakravorti21 <sc...@bloomberg.net>
Co-authored-by: SChakravorti21<sc...@bloomberg.net>
Co-authored-by: Jacob Wujciak-Jens <ja...@wujciak.de>
Signed-off-by: Felipe Oliveira Carvalho <fe...@gmail.com>
---
cpp/src/arrow/chunk_resolver.cc | 4 +--
cpp/src/arrow/chunk_resolver.h | 40 +++++++++++++---------
cpp/src/arrow/chunk_resolver_benchmark.cc | 3 --
cpp/src/arrow/chunked_array.h | 2 +-
cpp/src/arrow/chunked_array_test.cc | 4 ---
cpp/src/arrow/compute/kernels/chunked_internal.h | 2 +-
cpp/src/arrow/compute/kernels/vector_sort.cc | 3 +-
.../arrow/compute/kernels/vector_sort_internal.h | 4 +--
cpp/src/arrow/stl_iterator.h | 2 +-
docs/source/cpp/api/array.rst | 14 ++++++++
r/src/altrep.cpp | 14 +++++---
11 files changed, 55 insertions(+), 37 deletions(-)
diff --git a/cpp/src/arrow/chunk_resolver.cc b/cpp/src/arrow/chunk_resolver.cc
index bda6b17810..ca74ffa06c 100644
--- a/cpp/src/arrow/chunk_resolver.cc
+++ b/cpp/src/arrow/chunk_resolver.cc
@@ -26,7 +26,7 @@
#include "arrow/array.h"
#include "arrow/record_batch.h"
-namespace arrow::internal {
+namespace arrow {
namespace {
template <typename T>
@@ -167,4 +167,4 @@ void ChunkResolver::ResolveManyImpl(int64_t n_indices, const uint64_t* logical_i
logical_index_vec, out_chunk_location_vec, chunk_hint);
}
-} // namespace arrow::internal
+} // namespace arrow
diff --git a/cpp/src/arrow/chunk_resolver.h b/cpp/src/arrow/chunk_resolver.h
index 4a5e27c053..ab0e753d00 100644
--- a/cpp/src/arrow/chunk_resolver.h
+++ b/cpp/src/arrow/chunk_resolver.h
@@ -27,12 +27,12 @@
#include "arrow/type_fwd.h"
#include "arrow/util/macros.h"
-namespace arrow::internal {
+namespace arrow {
-struct ChunkResolver;
+class ChunkResolver;
template <typename IndexType>
-struct TypedChunkLocation {
+struct ARROW_EXPORT TypedChunkLocation {
/// \brief Index of the chunk in the array of chunks
///
/// The value is always in the range `[0, chunks.size()]`. `chunks.size()` is used
@@ -41,7 +41,7 @@ struct TypedChunkLocation {
/// \brief Index of the value in the chunk
///
- /// The value is UNDEFINED if chunk_index >= chunks.size()
+ /// The value is UNDEFINED if `chunk_index >= chunks.size()`
IndexType index_in_chunk = 0;
TypedChunkLocation() = default;
@@ -61,7 +61,7 @@ using ChunkLocation = TypedChunkLocation<int64_t>;
/// \brief An utility that incrementally resolves logical indices into
/// physical indices in a chunked array.
-struct ARROW_EXPORT ChunkResolver {
+class ARROW_EXPORT ChunkResolver {
private:
/// \brief Array containing `chunks.size() + 1` offsets.
///
@@ -76,14 +76,11 @@ struct ARROW_EXPORT ChunkResolver {
public:
explicit ChunkResolver(const ArrayVector& chunks) noexcept;
+
explicit ChunkResolver(const std::vector<const Array*>& chunks) noexcept;
+
explicit ChunkResolver(const RecordBatchVector& batches) noexcept;
- /// \brief Construct a ChunkResolver from a vector of chunks.size() + 1 offsets.
- ///
- /// The first offset must be 0 and the last offset must be the logical length of the
- /// chunked array. Each offset before the last represents the starting logical index of
- /// the corresponding chunk.
explicit ChunkResolver(std::vector<int64_t> offsets) noexcept
: offsets_(std::move(offsets)), cached_chunk_(0) {
#ifndef NDEBUG
@@ -115,11 +112,11 @@ struct ARROW_EXPORT ChunkResolver {
/// The returned ChunkLocation contains the chunk index and the within-chunk index
/// equivalent to the logical index.
///
- /// \pre index >= 0
- /// \post location.chunk_index in [0, chunks.size()]
+ /// \pre `index >= 0`
+ /// \post `location.chunk_index` in `[0, chunks.size()]`
/// \param index The logical index to resolve
/// \return ChunkLocation with a valid chunk_index if index is within
- /// bounds, or with chunk_index == chunks.size() if logical index is
+ /// bounds, or with `chunk_index == chunks.size()` if logical index is
/// `>= chunked_array.length()`.
inline ChunkLocation Resolve(int64_t index) const {
const auto cached_chunk = cached_chunk_.load(std::memory_order_relaxed);
@@ -133,13 +130,13 @@ struct ARROW_EXPORT ChunkResolver {
/// The returned ChunkLocation contains the chunk index and the within-chunk index
/// equivalent to the logical index.
///
- /// \pre index >= 0
- /// \post location.chunk_index in [0, chunks.size()]
+ /// \pre `index >= 0`
+ /// \post `location.chunk_index` in `[0, chunks.size()]`
/// \param index The logical index to resolve
/// \param hint ChunkLocation{} or the last ChunkLocation returned by
/// this ChunkResolver.
/// \return ChunkLocation with a valid chunk_index if index is within
- /// bounds, or with chunk_index == chunks.size() if logical index is
+ /// bounds, or with `chunk_index == chunks.size()` if logical index is
/// `>= chunked_array.length()`.
inline ChunkLocation ResolveWithHint(int64_t index, ChunkLocation hint) const {
assert(hint.chunk_index < static_cast<uint32_t>(offsets_.size()));
@@ -281,4 +278,13 @@ struct ARROW_EXPORT ChunkResolver {
}
};
-} // namespace arrow::internal
+// Explicitly instantiate template base struct, for DLL linking on Windows
+template struct TypedChunkLocation<int32_t>;
+template struct TypedChunkLocation<int16_t>;
+template struct TypedChunkLocation<int8_t>;
+template struct TypedChunkLocation<uint8_t>;
+template struct TypedChunkLocation<uint16_t>;
+template struct TypedChunkLocation<uint32_t>;
+template struct TypedChunkLocation<int64_t>;
+template struct TypedChunkLocation<uint64_t>;
+} // namespace arrow
diff --git a/cpp/src/arrow/chunk_resolver_benchmark.cc b/cpp/src/arrow/chunk_resolver_benchmark.cc
index 0756de3fbe..a6f539a444 100644
--- a/cpp/src/arrow/chunk_resolver_benchmark.cc
+++ b/cpp/src/arrow/chunk_resolver_benchmark.cc
@@ -28,9 +28,6 @@
namespace arrow {
-using internal::ChunkResolver;
-using internal::TypedChunkLocation;
-
namespace {
int64_t constexpr kChunkedArrayLength = std::numeric_limits<uint16_t>::max();
diff --git a/cpp/src/arrow/chunked_array.h b/cpp/src/arrow/chunked_array.h
index c65b6cb6e2..02bcd0f902 100644
--- a/cpp/src/arrow/chunked_array.h
+++ b/cpp/src/arrow/chunked_array.h
@@ -199,7 +199,7 @@ class ARROW_EXPORT ChunkedArray {
private:
template <typename T, typename V>
friend class ::arrow::stl::ChunkedArrayIterator;
- internal::ChunkResolver chunk_resolver_;
+ ChunkResolver chunk_resolver_;
ARROW_DISALLOW_COPY_AND_ASSIGN(ChunkedArray);
};
diff --git a/cpp/src/arrow/chunked_array_test.cc b/cpp/src/arrow/chunked_array_test.cc
index f98dde689c..b3944fd1b1 100644
--- a/cpp/src/arrow/chunked_array_test.cc
+++ b/cpp/src/arrow/chunked_array_test.cc
@@ -36,10 +36,6 @@
namespace arrow {
-using internal::ChunkLocation;
-using internal::ChunkResolver;
-using internal::TypedChunkLocation;
-
class TestChunkedArray : public ::testing::Test {
protected:
virtual void Construct() {
diff --git a/cpp/src/arrow/compute/kernels/chunked_internal.h b/cpp/src/arrow/compute/kernels/chunked_internal.h
index 2b72e0ab31..f7cb615f3e 100644
--- a/cpp/src/arrow/compute/kernels/chunked_internal.h
+++ b/cpp/src/arrow/compute/kernels/chunked_internal.h
@@ -52,7 +52,7 @@ struct ResolvedChunk {
class ChunkedArrayResolver {
private:
- ::arrow::internal::ChunkResolver resolver_;
+ ChunkResolver resolver_;
std::vector<const Array*> chunks_;
public:
diff --git a/cpp/src/arrow/compute/kernels/vector_sort.cc b/cpp/src/arrow/compute/kernels/vector_sort.cc
index 8766ca3baa..395ed86a06 100644
--- a/cpp/src/arrow/compute/kernels/vector_sort.cc
+++ b/cpp/src/arrow/compute/kernels/vector_sort.cc
@@ -24,7 +24,6 @@
namespace arrow {
using internal::checked_cast;
-using internal::ChunkLocation;
namespace compute {
namespace internal {
@@ -852,7 +851,7 @@ class TableSorter {
const RecordBatchVector batches_;
const SortOptions& options_;
const NullPlacement null_placement_;
- const ::arrow::internal::ChunkResolver left_resolver_, right_resolver_;
+ const ::arrow::ChunkResolver left_resolver_, right_resolver_;
const std::vector<ResolvedSortKey> sort_keys_;
uint64_t* indices_begin_;
uint64_t* indices_end_;
diff --git a/cpp/src/arrow/compute/kernels/vector_sort_internal.h b/cpp/src/arrow/compute/kernels/vector_sort_internal.h
index 564afb8c08..bee7f838a0 100644
--- a/cpp/src/arrow/compute/kernels/vector_sort_internal.h
+++ b/cpp/src/arrow/compute/kernels/vector_sort_internal.h
@@ -749,9 +749,9 @@ struct ResolvedTableSortKey {
order(order),
null_count(null_count) {}
- using LocationType = ::arrow::internal::ChunkLocation;
+ using LocationType = ::arrow::ChunkLocation;
- ResolvedChunk GetChunk(::arrow::internal::ChunkLocation loc) const {
+ ResolvedChunk GetChunk(::arrow::ChunkLocation loc) const {
return {chunks[loc.chunk_index], loc.index_in_chunk};
}
diff --git a/cpp/src/arrow/stl_iterator.h b/cpp/src/arrow/stl_iterator.h
index 5f2acfb071..577066cba0 100644
--- a/cpp/src/arrow/stl_iterator.h
+++ b/cpp/src/arrow/stl_iterator.h
@@ -237,7 +237,7 @@ class ChunkedArrayIterator {
}
private:
- arrow::internal::ChunkLocation GetChunkLocation(int64_t index) const {
+ arrow::ChunkLocation GetChunkLocation(int64_t index) const {
assert(chunked_array_);
return chunked_array_->chunk_resolver_.Resolve(index);
}
diff --git a/docs/source/cpp/api/array.rst b/docs/source/cpp/api/array.rst
index a7e5d0cf07..133a432884 100644
--- a/docs/source/cpp/api/array.rst
+++ b/docs/source/cpp/api/array.rst
@@ -19,6 +19,9 @@
Arrays
======
+Base classes
+============
+
.. doxygenclass:: arrow::ArrayData
:project: arrow_cpp
:members:
@@ -85,6 +88,17 @@ Chunked Arrays
:project: arrow_cpp
:members:
+.. doxygenstruct:: arrow::ChunkLocation
+ :project: arrow_cpp
+ :members:
+
+.. doxygenstruct:: arrow::TypedChunkLocation
+ :project: arrow_cpp
+ :members:
+
+.. doxygenclass:: arrow::ChunkResolver
+ :project: arrow_cpp
+ :members:
Utilities
=========
diff --git a/r/src/altrep.cpp b/r/src/altrep.cpp
index bdaac0a9ce..90a459e19c 100644
--- a/r/src/altrep.cpp
+++ b/r/src/altrep.cpp
@@ -80,6 +80,14 @@ void DeletePointer(std::shared_ptr<T>* ptr) {
template <typename T>
using Pointer = cpp11::external_pointer<std::shared_ptr<T>, DeletePointer<T>>;
+#if ARROW_VERSION_MAJOR >= 18
+using ChunkResolver = arrow::ChunkResolver;
+using ChunkLocation = arrow::ChunkLocation;
+#else
+using ChunkResolver = arrow::internal::ChunkResolver;
+using ChunkLocation = arrow::internal::ChunkLocation;
+#endif
+
class ArrowAltrepData {
public:
explicit ArrowAltrepData(const std::shared_ptr<ChunkedArray>& chunked_array)
@@ -87,13 +95,11 @@ class ArrowAltrepData {
const std::shared_ptr<ChunkedArray>& chunked_array() { return chunked_array_; }
- arrow::internal::ChunkLocation locate(int64_t index) {
- return resolver_.Resolve(index);
- }
+ ChunkLocation locate(int64_t index) { return resolver_.Resolve(index); }
private:
std::shared_ptr<ChunkedArray> chunked_array_;
- arrow::internal::ChunkResolver resolver_;
+ ChunkResolver resolver_;
};
// the ChunkedArray that is being wrapped by the altrep object
(arrow) 02/15: GH-44361: [C#][Integration] Include .NET in Flight integration tests (#44377)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit f42c5b0c240d7275ec28081aa9be71f17a61e3bf
Author: Adam Reeve <ad...@gmail.com>
AuthorDate: Wed Oct 16 03:09:41 2024 +1300
GH-44361: [C#][Integration] Include .NET in Flight integration tests (#44377)
### Rationale for this change
See #44361. This allows testing compatibility of the .NET Flight implementation with other Flight implementations.
### What changes are included in this PR?
* Adds a new `Apache.Arrow.Flight.IntegrationTest` project that can run in server or client mode for Flight integration tests.
* Includes the integration tests that send then retrieve data defined in JSON files, but doesn't add any of the named scenarios
* Configures archery to include C# in the Flight integration tests, but skip all the named scenarios
* Also skips tests that use dictionary data due to #38045, and the empty data test due to #44363
### Are these changes tested?
These changes are tests.
### Are there any user-facing changes?
No
* GitHub Issue: #44361
Authored-by: Adam Reeve <ad...@gmail.com>
Signed-off-by: Curt Hagenlocher <cu...@hagenlocher.org>
---
csharp/Apache.Arrow.sln | 6 +
.../FlightRecordBatchStreamWriter.cs | 2 +-
.../Apache.Arrow.Flight.IntegrationTest.csproj | 18 +++
.../FlightClientCommand.cs | 51 +++++++
.../FlightServerCommand.cs | 68 +++++++++
.../GrpcResolver.cs | 34 +++++
.../JsonTestScenario.cs | 167 +++++++++++++++++++++
.../Apache.Arrow.Flight.IntegrationTest/Program.cs | 67 +++++++++
csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs | 11 +-
.../TestFlightServer.cs | 8 +-
dev/archery/archery/integration/datagen.py | 23 ++-
dev/archery/archery/integration/runner.py | 9 +-
dev/archery/archery/integration/tester_csharp.py | 57 ++++++-
dev/release/rat_exclude_files.txt | 1 +
14 files changed, 502 insertions(+), 20 deletions(-)
diff --git a/csharp/Apache.Arrow.sln b/csharp/Apache.Arrow.sln
index 7e7f7c6331..0e569de1d6 100644
--- a/csharp/Apache.Arrow.sln
+++ b/csharp/Apache.Arrow.sln
@@ -27,6 +27,8 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Apache.Arrow.Flight.Sql.Tes
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Apache.Arrow.Flight.Sql", "src\Apache.Arrow.Flight.Sql\Apache.Arrow.Flight.Sql.csproj", "{2ADE087A-B424-4895-8CC5-10170D10BA62}"
EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Apache.Arrow.Flight.IntegrationTest", "test\Apache.Arrow.Flight.IntegrationTest\Apache.Arrow.Flight.IntegrationTest.csproj", "{7E66CBB4-D921-41E7-A98A-7C6DEA521696}"
+EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
@@ -81,6 +83,10 @@ Global
{2ADE087A-B424-4895-8CC5-10170D10BA62}.Debug|Any CPU.Build.0 = Debug|Any CPU
{2ADE087A-B424-4895-8CC5-10170D10BA62}.Release|Any CPU.ActiveCfg = Release|Any CPU
{2ADE087A-B424-4895-8CC5-10170D10BA62}.Release|Any CPU.Build.0 = Release|Any CPU
+ {7E66CBB4-D921-41E7-A98A-7C6DEA521696}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+ {7E66CBB4-D921-41E7-A98A-7C6DEA521696}.Debug|Any CPU.Build.0 = Debug|Any CPU
+ {7E66CBB4-D921-41E7-A98A-7C6DEA521696}.Release|Any CPU.ActiveCfg = Release|Any CPU
+ {7E66CBB4-D921-41E7-A98A-7C6DEA521696}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
diff --git a/csharp/src/Apache.Arrow.Flight/FlightRecordBatchStreamWriter.cs b/csharp/src/Apache.Arrow.Flight/FlightRecordBatchStreamWriter.cs
index f76f082245..7a8a6fd677 100644
--- a/csharp/src/Apache.Arrow.Flight/FlightRecordBatchStreamWriter.cs
+++ b/csharp/src/Apache.Arrow.Flight/FlightRecordBatchStreamWriter.cs
@@ -64,7 +64,7 @@ namespace Apache.Arrow.Flight
{
if (!_disposed)
{
- _flightDataStream.Dispose();
+ _flightDataStream?.Dispose();
_disposed = true;
}
}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/Apache.Arrow.Flight.IntegrationTest.csproj b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Apache.Arrow.Flight.IntegrationTest.csproj
new file mode 100644
index 0000000000..34030621b4
--- /dev/null
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Apache.Arrow.Flight.IntegrationTest.csproj
@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project Sdk="Microsoft.NET.Sdk">
+
+ <PropertyGroup>
+ <OutputType>Exe</OutputType>
+ <TargetFramework>net8.0</TargetFramework>
+ <RootNamespace>Apache.Arrow.Flight.IntegrationTest</RootNamespace>
+ </PropertyGroup>
+
+ <ItemGroup>
+ <PackageReference Include="System.CommandLine" Version="2.0.0-beta4.22272.1" />
+ <PackageReference Include="System.Text.Json" Version="8.0.5" />
+ <ProjectReference Include="..\..\src\Apache.Arrow.Flight\Apache.Arrow.Flight.csproj" />
+ <ProjectReference Include="..\Apache.Arrow.Flight.TestWeb\Apache.Arrow.Flight.TestWeb.csproj" />
+ <ProjectReference Include="..\Apache.Arrow.IntegrationTest\Apache.Arrow.IntegrationTest.csproj" />
+ </ItemGroup>
+
+</Project>
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs
new file mode 100644
index 0000000000..d9e0ff5230
--- /dev/null
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.IO;
+using System.Threading.Tasks;
+
+namespace Apache.Arrow.Flight.IntegrationTest;
+
+public class FlightClientCommand
+{
+ private readonly int _port;
+ private readonly string _scenario;
+ private readonly FileInfo _jsonFileInfo;
+
+ public FlightClientCommand(int port, string scenario, FileInfo jsonFileInfo)
+ {
+ _port = port;
+ _scenario = scenario;
+ _jsonFileInfo = jsonFileInfo;
+ }
+
+ public async Task Execute()
+ {
+ if (!string.IsNullOrEmpty(_scenario))
+ {
+ // No named scenarios are currently implemented
+ throw new Exception($"Scenario '{_scenario}' is not supported.");
+ }
+
+ if (!(_jsonFileInfo?.Exists ?? false))
+ {
+ throw new Exception($"Invalid JSON file path '{_jsonFileInfo?.FullName}'");
+ }
+
+ var scenario = new JsonTestScenario(_port, _jsonFileInfo);
+ await scenario.RunClient().ConfigureAwait(false);
+ }
+}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightServerCommand.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightServerCommand.cs
new file mode 100644
index 0000000000..c3a7694485
--- /dev/null
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightServerCommand.cs
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Net;
+using System.Threading.Tasks;
+using Apache.Arrow.Flight.TestWeb;
+using Microsoft.AspNetCore.Hosting;
+using Microsoft.AspNetCore.Hosting.Server;
+using Microsoft.AspNetCore.Hosting.Server.Features;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+
+namespace Apache.Arrow.Flight.IntegrationTest;
+
+public class FlightServerCommand
+{
+ private readonly string _scenario;
+
+ public FlightServerCommand(string scenario)
+ {
+ _scenario = scenario;
+ }
+
+ public async Task Execute()
+ {
+ if (!string.IsNullOrEmpty(_scenario))
+ {
+ // No named scenarios are currently implemented
+ throw new Exception($"Scenario '{_scenario}' is not supported.");
+ }
+
+ var host = Host.CreateDefaultBuilder()
+ .ConfigureWebHostDefaults(webBuilder =>
+ {
+ webBuilder
+ .ConfigureKestrel(options =>
+ {
+ options.Listen(IPEndPoint.Parse("127.0.0.1:0"), l => l.Protocols = HttpProtocols.Http2);
+ })
+ .UseStartup<Startup>();
+ })
+ .Build();
+
+ await host.StartAsync().ConfigureAwait(false);
+
+ var addresses = host.Services.GetService<IServer>().Features.Get<IServerAddressesFeature>().Addresses;
+ foreach (var address in addresses)
+ {
+ Console.WriteLine($"Server listening on {address}");
+ }
+
+ await host.WaitForShutdownAsync().ConfigureAwait(false);
+ }
+}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/GrpcResolver.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/GrpcResolver.cs
new file mode 100644
index 0000000000..44b1075e7a
--- /dev/null
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/GrpcResolver.cs
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using Grpc.Net.Client.Balancer;
+
+namespace Apache.Arrow.Flight.IntegrationTest;
+
+/// <summary>
+/// The Grpc.Net.Client library doesn't know how to handle the "grpc+tcp" scheme used by Arrow Flight.
+/// This ResolverFactory passes these through to the standard Static Resolver used for the http scheme.
+/// </summary>
+public class GrpcTcpResolverFactory : ResolverFactory
+{
+ public override string Name => "grpc+tcp";
+
+ public override Resolver Create(ResolverOptions options)
+ {
+ return new StaticResolverFactory(
+ uri => new[] { new BalancerAddress(options.Address.Host, options.Address.Port) })
+ .Create(options);
+ }
+}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/JsonTestScenario.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/JsonTestScenario.cs
new file mode 100644
index 0000000000..f4f3ac28bf
--- /dev/null
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/JsonTestScenario.cs
@@ -0,0 +1,167 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.IO;
+using System.Linq;
+using System.Threading.Tasks;
+using Apache.Arrow.Flight.Client;
+using Apache.Arrow.IntegrationTest;
+using Apache.Arrow.Tests;
+using Apache.Arrow.Types;
+using Google.Protobuf;
+using Grpc.Net.Client;
+using Grpc.Core;
+using Grpc.Net.Client.Balancer;
+using Microsoft.Extensions.DependencyInjection;
+
+namespace Apache.Arrow.Flight.IntegrationTest;
+
+/// <summary>
+/// A test scenario defined using a JSON data file
+/// </summary>
+internal class JsonTestScenario
+{
+ private readonly int _serverPort;
+ private readonly FileInfo _jsonFile;
+ private readonly ServiceProvider _serviceProvider;
+
+ public JsonTestScenario(int serverPort, FileInfo jsonFile)
+ {
+ _serverPort = serverPort;
+ _jsonFile = jsonFile;
+
+ var services = new ServiceCollection();
+ services.AddSingleton<ResolverFactory>(new GrpcTcpResolverFactory());
+ _serviceProvider = services.BuildServiceProvider();
+ }
+
+ public async Task RunClient()
+ {
+ var address = $"grpc+tcp://localhost:{_serverPort}";
+ using var channel = GrpcChannel.ForAddress(
+ address,
+ new GrpcChannelOptions
+ {
+ ServiceProvider = _serviceProvider,
+ Credentials = ChannelCredentials.Insecure
+ });
+ var client = new FlightClient(channel);
+
+ var descriptor = FlightDescriptor.CreatePathDescriptor(_jsonFile.FullName);
+
+ var jsonFile = await JsonFile.ParseAsync(_jsonFile).ConfigureAwait(false);
+ var schema = jsonFile.GetSchemaAndDictionaries(out Func<DictionaryType, IArrowArray> dictionaries);
+ var batches = jsonFile.Batches.Select(batch => batch.ToArrow(schema, dictionaries)).ToArray();
+
+ // 1. Put the data to the server.
+ await UploadBatches(client, descriptor, batches).ConfigureAwait(false);
+
+ // 2. Get the ticket for the data.
+ var info = await client.GetInfo(descriptor).ConfigureAwait(false);
+ if (info.Endpoints.Count == 0)
+ {
+ throw new Exception("No endpoints received");
+ }
+
+ // 3. Stream data from the server, comparing individual batches.
+ foreach (var endpoint in info.Endpoints)
+ {
+ var locations = endpoint.Locations.ToArray();
+ if (locations.Length == 0)
+ {
+ // Can read with existing client
+ await ConsumeFlightLocation(client, endpoint.Ticket, batches).ConfigureAwait(false);
+ }
+ else
+ {
+ foreach (var location in locations)
+ {
+ using var readChannel = GrpcChannel.ForAddress(
+ location.Uri,
+ new GrpcChannelOptions
+ {
+ ServiceProvider = _serviceProvider,
+ Credentials = ChannelCredentials.Insecure
+ });
+ var readClient = new FlightClient(readChannel);
+ await ConsumeFlightLocation(readClient, endpoint.Ticket, batches).ConfigureAwait(false);
+ }
+ }
+ }
+ }
+
+ private static async Task UploadBatches(FlightClient client, FlightDescriptor descriptor, RecordBatch[] batches)
+ {
+ using var putCall = client.StartPut(descriptor);
+ using var writer = putCall.RequestStream;
+
+ try
+ {
+ var counter = 0;
+ foreach (var batch in batches)
+ {
+ var metadata = $"{counter}";
+
+ await writer.WriteAsync(batch, ByteString.CopyFromUtf8(metadata)).ConfigureAwait(false);
+
+ // Verify server has acknowledged the write request
+ await putCall.ResponseStream.MoveNext().ConfigureAwait(false);
+ var responseString = putCall.ResponseStream.Current.ApplicationMetadata.ToStringUtf8();
+
+ if (responseString != metadata)
+ {
+ throw new Exception($"Response metadata '{responseString}' does not match expected metadata '{metadata}'");
+ }
+
+ counter++;
+ }
+ }
+ finally
+ {
+ await writer.CompleteAsync().ConfigureAwait(false);
+ }
+
+ // Drain the response stream to ensure the server has stored the data
+ var hasMore = await putCall.ResponseStream.MoveNext().ConfigureAwait(false);
+ if (hasMore)
+ {
+ throw new Exception("Expected to have reached the end of the response stream");
+ }
+ }
+
+ private static async Task ConsumeFlightLocation(FlightClient client, FlightTicket ticket, RecordBatch[] batches)
+ {
+ using var readStream = client.GetStream(ticket);
+ var counter = 0;
+ foreach (var originalBatch in batches)
+ {
+ if (!await readStream.ResponseStream.MoveNext().ConfigureAwait(false))
+ {
+ throw new Exception($"Expected {batches.Length} batches but received {counter}");
+ }
+
+ var batch = readStream.ResponseStream.Current;
+ ArrowReaderVerifier.CompareBatches(originalBatch, batch, strictCompare: false);
+
+ counter++;
+ }
+
+ if (await readStream.ResponseStream.MoveNext().ConfigureAwait(false))
+ {
+ throw new Exception($"Expected to reach the end of the response stream after {batches.Length} batches");
+ }
+ }
+}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/Program.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Program.cs
new file mode 100644
index 0000000000..24d39de28a
--- /dev/null
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Program.cs
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System.CommandLine;
+using System.IO;
+using System.Threading.Tasks;
+
+namespace Apache.Arrow.Flight.IntegrationTest;
+
+public static class Program
+{
+ public static async Task<int> Main(string[] args)
+ {
+ var portOption = new Option<int>(
+ new[] { "--port", "-p" },
+ description: "Port the Flight server is listening on");
+ var scenarioOption = new Option<string>(
+ new[] { "--scenario", "-s" },
+ "The name of the scenario to run");
+ var pathOption = new Option<FileInfo>(
+ new[] { "--path", "-j" },
+ "Path to a JSON file of test data");
+
+ var rootCommand = new RootCommand(
+ "Integration test application for Apache.Arrow .NET Flight.");
+
+ var clientCommand = new Command("client", "Run the Flight client")
+ {
+ portOption,
+ scenarioOption,
+ pathOption,
+ };
+ rootCommand.AddCommand(clientCommand);
+
+ clientCommand.SetHandler(async (port, scenario, jsonFile) =>
+ {
+ var command = new FlightClientCommand(port, scenario, jsonFile);
+ await command.Execute().ConfigureAwait(false);
+ }, portOption, scenarioOption, pathOption);
+
+ var serverCommand = new Command("server", "Run the Flight server")
+ {
+ scenarioOption,
+ };
+ rootCommand.AddCommand(serverCommand);
+
+ serverCommand.SetHandler(async scenario =>
+ {
+ var command = new FlightServerCommand(scenario);
+ await command.Execute().ConfigureAwait(false);
+ }, scenarioOption);
+
+ return await rootCommand.InvokeAsync(args).ConfigureAwait(false);
+ }
+}
diff --git a/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs b/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs
index 97c1af2f06..d1cfe9e445 100644
--- a/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs
+++ b/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs
@@ -13,15 +13,13 @@
// See the License for the specific language governing permissions and
// limitations under the License.
-using System;
-using System.Collections.Generic;
-using System.Linq;
-using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Logging.Console;
namespace Apache.Arrow.Flight.TestWeb
{
@@ -35,6 +33,11 @@ namespace Apache.Arrow.Flight.TestWeb
.AddFlightServer<TestFlightServer>();
services.AddSingleton(new FlightStore());
+
+ // The integration tests rely on the port being written to the first line of stdout,
+ // so send all logging to stderr.
+ services.Configure<ConsoleLoggerOptions>(
+ o => o.LogToStandardErrorThreshold = LogLevel.Debug);
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
diff --git a/csharp/test/Apache.Arrow.Flight.TestWeb/TestFlightServer.cs b/csharp/test/Apache.Arrow.Flight.TestWeb/TestFlightServer.cs
index 4a72b73274..46c5460912 100644
--- a/csharp/test/Apache.Arrow.Flight.TestWeb/TestFlightServer.cs
+++ b/csharp/test/Apache.Arrow.Flight.TestWeb/TestFlightServer.cs
@@ -67,14 +67,16 @@ namespace Apache.Arrow.Flight.TestWeb
if(!_flightStore.Flights.TryGetValue(flightDescriptor, out var flightHolder))
{
- flightHolder = new FlightHolder(flightDescriptor, await requestStream.Schema, $"http://{context.Host}");
+ flightHolder = new FlightHolder(flightDescriptor, await requestStream.Schema, $"grpc+tcp://{context.Host}");
_flightStore.Flights.Add(flightDescriptor, flightHolder);
}
while (await requestStream.MoveNext())
{
- flightHolder.AddBatch(new RecordBatchWithMetadata(requestStream.Current, requestStream.ApplicationMetadata.FirstOrDefault()));
- await responseStream.WriteAsync(FlightPutResult.Empty);
+ var applicationMetadata = requestStream.ApplicationMetadata.FirstOrDefault();
+ flightHolder.AddBatch(new RecordBatchWithMetadata(requestStream.Current, applicationMetadata));
+ await responseStream.WriteAsync(
+ applicationMetadata == null ? FlightPutResult.Empty : new FlightPutResult(applicationMetadata));
}
}
diff --git a/dev/archery/archery/integration/datagen.py b/dev/archery/archery/integration/datagen.py
index 9f86d172dd..bc86296340 100644
--- a/dev/archery/archery/integration/datagen.py
+++ b/dev/archery/archery/integration/datagen.py
@@ -25,7 +25,7 @@ import tempfile
import numpy as np
from .util import frombytes, tobytes, random_bytes, random_utf8
-from .util import SKIP_C_SCHEMA, SKIP_C_ARRAY
+from .util import SKIP_C_SCHEMA, SKIP_C_ARRAY, SKIP_FLIGHT
def metadata_key_values(pairs):
@@ -1890,7 +1890,10 @@ def get_generated_json_files(tempdir=None):
return
file_objs = [
- generate_primitive_case([], name='primitive_no_batches'),
+ generate_primitive_case([], name='primitive_no_batches')
+ # TODO(https://github.com/apache/arrow/issues/44363)
+ .skip_format(SKIP_FLIGHT, 'C#'),
+
generate_primitive_case([17, 20], name='primitive'),
generate_primitive_case([0, 0, 0], name='primitive_zerolength'),
@@ -1952,16 +1955,22 @@ def get_generated_json_files(tempdir=None):
generate_dictionary_case()
# TODO(https://github.com/apache/arrow-nanoarrow/issues/622)
- .skip_tester('nanoarrow'),
+ .skip_tester('nanoarrow')
+ # TODO(https://github.com/apache/arrow/issues/38045)
+ .skip_format(SKIP_FLIGHT, 'C#'),
generate_dictionary_unsigned_case()
.skip_tester('nanoarrow')
- .skip_tester('Java'), # TODO(ARROW-9377)
+ .skip_tester('Java') # TODO(ARROW-9377)
+ # TODO(https://github.com/apache/arrow/issues/38045)
+ .skip_format(SKIP_FLIGHT, 'C#'),
generate_nested_dictionary_case()
# TODO(https://github.com/apache/arrow-nanoarrow/issues/622)
.skip_tester('nanoarrow')
- .skip_tester('Java'), # TODO(ARROW-7779)
+ .skip_tester('Java') # TODO(ARROW-7779)
+ # TODO(https://github.com/apache/arrow/issues/38045)
+ .skip_format(SKIP_FLIGHT, 'C#'),
generate_run_end_encoded_case()
.skip_tester('C#')
@@ -1988,7 +1997,9 @@ def get_generated_json_files(tempdir=None):
.skip_tester('nanoarrow')
# TODO: ensure the extension is registered in the C++ entrypoint
.skip_format(SKIP_C_SCHEMA, 'C++')
- .skip_format(SKIP_C_ARRAY, 'C++'),
+ .skip_format(SKIP_C_ARRAY, 'C++')
+ # TODO(https://github.com/apache/arrow/issues/38045)
+ .skip_format(SKIP_FLIGHT, 'C#'),
]
generated_paths = []
diff --git a/dev/archery/archery/integration/runner.py b/dev/archery/archery/integration/runner.py
index e276738846..378b17d75f 100644
--- a/dev/archery/archery/integration/runner.py
+++ b/dev/archery/archery/integration/runner.py
@@ -631,10 +631,13 @@ def run_all_tests(with_cpp=True, with_java=True, with_js=True,
flight_scenarios = [
Scenario(
"auth:basic_proto",
- description="Authenticate using the BasicAuth protobuf."),
+ description="Authenticate using the BasicAuth protobuf.",
+ skip_testers={"C#"},
+ ),
Scenario(
"middleware",
description="Ensure headers are propagated via middleware.",
+ skip_testers={"C#"},
),
Scenario(
"ordered",
@@ -689,12 +692,12 @@ def run_all_tests(with_cpp=True, with_java=True, with_js=True,
Scenario(
"flight_sql",
description="Ensure Flight SQL protocol is working as expected.",
- skip_testers={"Rust"}
+ skip_testers={"Rust", "C#"}
),
Scenario(
"flight_sql:extension",
description="Ensure Flight SQL extensions work as expected.",
- skip_testers={"Rust"}
+ skip_testers={"Rust", "C#"}
),
Scenario(
"flight_sql:ingestion",
diff --git a/dev/archery/archery/integration/tester_csharp.py b/dev/archery/archery/integration/tester_csharp.py
index 02ced0701d..50b3499fbf 100644
--- a/dev/archery/archery/integration/tester_csharp.py
+++ b/dev/archery/archery/integration/tester_csharp.py
@@ -17,6 +17,7 @@
from contextlib import contextmanager
import os
+import subprocess
from . import cdata
from .tester import Tester, CDataExporter, CDataImporter
@@ -25,12 +26,20 @@ from ..utils.source import ARROW_ROOT_DEFAULT
_ARTIFACTS_PATH = os.path.join(ARROW_ROOT_DEFAULT, "csharp/artifacts")
+_BUILD_SUBDIR = "Debug/net8.0"
_EXE_PATH = os.path.join(_ARTIFACTS_PATH,
"Apache.Arrow.IntegrationTest",
- "Debug/net8.0/Apache.Arrow.IntegrationTest",
+ _BUILD_SUBDIR,
+ "Apache.Arrow.IntegrationTest",
)
+_FLIGHT_EXE_PATH = os.path.join(_ARTIFACTS_PATH,
+ "Apache.Arrow.Flight.IntegrationTest",
+ _BUILD_SUBDIR,
+ "Apache.Arrow.Flight.IntegrationTest",
+ )
+
_clr_loaded = False
@@ -44,10 +53,10 @@ def _load_clr():
import clr
clr.AddReference(
f"{_ARTIFACTS_PATH}/Apache.Arrow.IntegrationTest/"
- f"Debug/net8.0/Apache.Arrow.IntegrationTest.dll")
+ f"{_BUILD_SUBDIR}/Apache.Arrow.IntegrationTest.dll")
clr.AddReference(
f"{_ARTIFACTS_PATH}/Apache.Arrow.Tests/"
- f"Debug/net8.0/Apache.Arrow.Tests.dll")
+ f"{_BUILD_SUBDIR}/Apache.Arrow.Tests.dll")
from Apache.Arrow.IntegrationTest import CDataInterface
CDataInterface.Initialize()
@@ -146,6 +155,8 @@ class CSharpCDataImporter(CDataImporter, _CDataBase):
class CSharpTester(Tester):
PRODUCER = True
CONSUMER = True
+ FLIGHT_SERVER = True
+ FLIGHT_CLIENT = True
C_DATA_SCHEMA_EXPORTER = True
C_DATA_SCHEMA_IMPORTER = True
C_DATA_ARRAY_EXPORTER = True
@@ -192,3 +203,43 @@ class CSharpTester(Tester):
def make_c_data_importer(self):
return CSharpCDataImporter(self.debug, self.args)
+
+ def flight_request(self, port, json_path=None, scenario_name=None):
+ cmd = [_FLIGHT_EXE_PATH, 'client', '--port', f'{port}']
+ if json_path:
+ cmd.extend(['--path', json_path])
+ elif scenario_name:
+ cmd.extend(['--scenario', scenario_name])
+ else:
+ raise TypeError("Must provide one of json_path or scenario_name")
+
+ if self.debug:
+ log(' '.join(cmd))
+ run_cmd(cmd)
+
+ @contextmanager
+ def flight_server(self, scenario_name=None):
+ cmd = [_FLIGHT_EXE_PATH, 'server']
+ if scenario_name:
+ cmd.extend(['--scenario', scenario_name])
+ if self.debug:
+ log(' '.join(cmd))
+ server = subprocess.Popen(
+ cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ try:
+ output = server.stdout.readline().decode()
+ if not output.startswith("Server listening on "):
+ server.kill()
+ out, err = server.communicate()
+ raise RuntimeError(
+ '.NET Flight server did not start properly, '
+ 'stdout: \n{}\n\nstderr:\n{}\n'.format(
+ output + out.decode(), err.decode()
+ )
+ )
+ port = int(output.split(':')[-1])
+ yield port
+ finally:
+ server.kill()
+ server.wait(5)
diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt
index e149c17981..dda1d36dc1 100644
--- a/dev/release/rat_exclude_files.txt
+++ b/dev/release/rat_exclude_files.txt
@@ -116,6 +116,7 @@ csharp/src/Apache.Arrow.Flight.AspNetCore/Apache.Arrow.Flight.AspNetCore.csproj
csharp/src/Apache.Arrow.Compression/Apache.Arrow.Compression.csproj
csharp/src/Apache.Arrow.Flight.Sql/Apache.Arrow.Flight.Sql.csproj
csharp/test/Apache.Arrow.Benchmarks/Apache.Arrow.Benchmarks.csproj
+csharp/test/Apache.Arrow.Flight.IntegrationTest/Apache.Arrow.Flight.IntegrationTest.csproj
csharp/test/Apache.Arrow.Flight.Tests/Apache.Arrow.Flight.Tests.csproj
csharp/test/Apache.Arrow.Flight.Sql.Tests/Apache.Arrow.Flight.Sql.Tests.csproj
csharp/test/Apache.Arrow.Flight.TestWeb/Apache.Arrow.Flight.TestWeb.csproj
(arrow) 14/15: GH-44674: [R] Fix R CMD check failure with dev testthat (#44675)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit d20173f488a2aabbc22fee2fc08c7e8461b3b5ae
Author: Bryce Mecum <pe...@gmail.com>
AuthorDate: Fri Nov 8 07:26:17 2024 -0800
GH-44674: [R] Fix R CMD check failure with dev testthat (#44675)
### Rationale for this change
Update assertion code in some of the R tests to be compatible with the in-development version of testthat.
### What changes are included in this PR?
Updated assertions in tests.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
Fixes https://github.com/apache/arrow/issues/44674.
* GitHub Issue: #44674
Authored-by: Bryce Mecum <pe...@gmail.com>
Signed-off-by: Bryce Mecum <pe...@gmail.com>
---
r/tests/testthat/test-Array.R | 6 +++---
r/tests/testthat/test-extension.R | 2 +-
r/tests/testthat/test-parquet.R | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/r/tests/testthat/test-Array.R b/r/tests/testthat/test-Array.R
index 3e41b8b274..fc8434945f 100644
--- a/r/tests/testthat/test-Array.R
+++ b/r/tests/testthat/test-Array.R
@@ -1261,7 +1261,7 @@ test_that("concat_arrays works", {
concat_int <- concat_arrays(arrow_array(1:3), arrow_array(4:5))
expect_true(concat_int$type == int32())
- expect_true(all(concat_int == arrow_array(1:5)))
+ expect_equal(concat_int, arrow_array(1:5))
concat_int64 <- concat_arrays(
arrow_array(1:3),
@@ -1269,7 +1269,7 @@ test_that("concat_arrays works", {
type = int64()
)
expect_true(concat_int64$type == int64())
- expect_true(all(concat_int == arrow_array(1:5)))
+ expect_equal(concat_int, arrow_array(1:5))
expect_error(
concat_arrays(
@@ -1283,7 +1283,7 @@ test_that("concat_arrays works", {
test_that("concat_arrays() coerces its input to Array", {
concat_ints <- concat_arrays(1L, 2L)
expect_true(concat_ints$type == int32())
- expect_true(all(concat_ints == arrow_array(c(1L, 2L))))
+ expect_equal(concat_ints, arrow_array(c(1L, 2L)))
expect_error(
concat_arrays(1L, "not a number", type = int32()),
diff --git a/r/tests/testthat/test-extension.R b/r/tests/testthat/test-extension.R
index db26a70acb..47ff1ad1a9 100644
--- a/r/tests/testthat/test-extension.R
+++ b/r/tests/testthat/test-extension.R
@@ -35,7 +35,7 @@ test_that("extension types can be created", {
expect_r6_class(array$type, "ExtensionType")
expect_true(array$type == type)
- expect_true(all(array$storage() == storage))
+ expect_equal(array$storage(), storage)
expect_identical(array$as_vector(), 1:10)
expect_identical(chunked_array(array)$as_vector(), 1:10)
diff --git a/r/tests/testthat/test-parquet.R b/r/tests/testthat/test-parquet.R
index 738febb7b3..edca48e92d 100644
--- a/r/tests/testthat/test-parquet.R
+++ b/r/tests/testthat/test-parquet.R
@@ -455,7 +455,7 @@ test_that("deprecated int96 timestamp unit can be specified when reading Parquet
)
expect_identical(result$some_datetime$type$unit(), TimeUnit$MILLI)
- expect_true(result$some_datetime == table$some_datetime)
+ expect_equal(result$some_datetime, table$some_datetime$cast(result$some_datetime$type))
})
test_that("Can read parquet with nested lists and maps", {
(arrow) 04/15: GH-44389: [Java][Integration][Release] Use Python 3.12 for verify-rc-source-integration-linux-conda-latest-amd64 (#44395)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit c13bed995fe9d95efddac8b4d476dd0c66dedf36
Author: Sutou Kouhei <ko...@clear-code.com>
AuthorDate: Wed Oct 16 18:39:30 2024 +0900
GH-44389: [Java][Integration][Release] Use Python 3.12 for verify-rc-source-integration-linux-conda-latest-amd64 (#44395)
### Rationale for this change
Because JPype doesn't support Python 3.13 yet.
See also:
* GH-44389
* https://discuss.python.org/t/api-for-python-3-13-prevents-use-of-3rd-party-gc-allocators/62709/5
### What changes are included in this PR?
Pin Python 3.12.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #44389
Lead-authored-by: Raúl Cumplido <ra...@gmail.com>
Co-authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Raúl Cumplido <ra...@gmail.com>
---
dev/tasks/verify-rc/github.linux.amd64.docker.yml | 9 +++++++++
dev/tasks/verify-rc/github.macos.yml | 2 +-
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/dev/tasks/verify-rc/github.linux.amd64.docker.yml b/dev/tasks/verify-rc/github.linux.amd64.docker.yml
index f2c0673314..97eecd2d28 100644
--- a/dev/tasks/verify-rc/github.linux.amd64.docker.yml
+++ b/dev/tasks/verify-rc/github.linux.amd64.docker.yml
@@ -41,6 +41,15 @@ jobs:
{% if distro == 'almalinux' and target|upper == 'PYTHON' %}
-e ARROW_GANDIVA=OFF \
{% endif %}
+ {% if distro == "conda" and target == "integration" %}
+ {# JPype doesn't work with Python 3.13.
+ # See also:
+ # * https://discuss.python.org/t/api-for-python-3-13-prevents-use-of-3rd-party-gc-allocators/62709/5
+ # * GH-44386
+ # * GH-44389
+ #}
+ -e PYTHON_VERSION="3.12" \
+ {% endif %}
-e VERIFY_RC="{{ rc|default("") }}" \
-e TEST_DEFAULT=0 \
-e TEST_{{ target|upper }}=1 \
diff --git a/dev/tasks/verify-rc/github.macos.yml b/dev/tasks/verify-rc/github.macos.yml
index 37a3e6f3fd..e0272e8f4e 100644
--- a/dev/tasks/verify-rc/github.macos.yml
+++ b/dev/tasks/verify-rc/github.macos.yml
@@ -65,7 +65,6 @@ jobs:
TEST_DEFAULT: 0
TEST_{{ target|upper }}: 1
{% if use_conda %}
- USE_CONDA: 1
{% if target == "integration" %}
# JPype doesn't work with Python 3.13.
# See also:
@@ -74,6 +73,7 @@ jobs:
# * GH-44389
PYTHON_VERSION: "3.12"
{% endif %}
+ USE_CONDA: 1
{% endif %}
run: |
arrow/dev/release/verify-release-candidate.sh {{ release|default("") }} {{ rc|default("") }}
(arrow) 15/15: GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN (#44685)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 2354c2f3866d5d73210586fb66ddde366d93f9f6
Author: Bryce Mecum <pe...@gmail.com>
AuthorDate: Mon Nov 11 12:29:37 2024 -0800
GH-44607: [C++][Dev] Update bundled Thrift, update mirrors to use CDN (#44685)
### Rationale for this change
Builds with bundled Thrift could fail because of permanently offline download mirror URLs. ASF now provides [a CDN](https://dlcdn.apache.org/).
### What changes are included in this PR?
1. Updates the pinned version for bundled Thrift from 0.16.0 to 0.20.0. We couldn't just update the URLs because the ASF CDN only hosts the last two versions of Thrift (0.20, 0.21.0). We didn't upgrade to Thrift 0.21.0 in this PR because it requires Boost 1.86.0 and we found incompatibilities with Boost 1.86.0 and `cpp/build-support/trim-boost.sh` which we use to publish a trimmed-down distribution of Boost for our build system to use.
2. Removes the set of URLs we were using to download Thrift in favor of just using the main CDN URL. We'll see how this works in practice.
### Are these changes tested?
Yes. I configured a build with `...-DThrift_SOURCE=BUNDLED...` and ran the full C++ test suite. I think further testing can be done in CI.
### Are there any user-facing changes?
No.
Closes https://github.com/apache/arrow/issues/44607
* GitHub Issue: #44607
Authored-by: Bryce Mecum <pe...@gmail.com>
Signed-off-by: Bryce Mecum <pe...@gmail.com>
---
cpp/cmake_modules/ThirdpartyToolchain.cmake | 19 ++++---------------
cpp/thirdparty/versions.txt | 6 +++---
2 files changed, 7 insertions(+), 18 deletions(-)
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index e07e13cab5..35ad4089e7 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -816,21 +816,9 @@ endif()
if(DEFINED ENV{ARROW_THRIFT_URL})
set(THRIFT_SOURCE_URL "$ENV{ARROW_THRIFT_URL}")
else()
- set_urls(THRIFT_SOURCE_URL
- "https://www.apache.org/dyn/closer.cgi?action=download&filename=/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://downloads.apache.org/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://apache.claz.org/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://apache.cs.utah.edu/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://apache.mirrors.lucidnetworks.net/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://apache.osuosl.org/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://ftp.wayne.edu/apache/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://mirror.olnevhost.net/pub/apache/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://mirrors.gigenet.com/apache/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://mirrors.koehn.com/apache/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://mirrors.ocf.berkeley.edu/apache/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://mirrors.sonic.net/apache/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "https://us.mirrors.quenda.co/apache/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
- "${THIRDPARTY_MIRROR_URL}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz")
+ set(THRIFT_SOURCE_URL
+ "https://dlcdn.apache.org/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
+ )
endif()
if(DEFINED ENV{ARROW_UCX_URL})
@@ -1775,6 +1763,7 @@ macro(build_thrift)
set(THRIFT_LIB_SUFFIX "md")
list(APPEND THRIFT_CMAKE_ARGS "-DWITH_MT=OFF")
endif()
+ # NOTE(amoeba): When you bump Thrift to >=0.21.0, change bin to lib
set(THRIFT_LIB
"${THRIFT_PREFIX}/bin/${CMAKE_IMPORT_LIBRARY_PREFIX}thrift${THRIFT_LIB_SUFFIX}${CMAKE_IMPORT_LIBRARY_SUFFIX}"
)
diff --git a/cpp/thirdparty/versions.txt b/cpp/thirdparty/versions.txt
index 30fa24a209..369a49744b 100644
--- a/cpp/thirdparty/versions.txt
+++ b/cpp/thirdparty/versions.txt
@@ -107,8 +107,8 @@ ARROW_SUBSTRAIT_BUILD_VERSION=v0.44.0
ARROW_SUBSTRAIT_BUILD_SHA256_CHECKSUM=f989a862f694e7dbb695925ddb7c4ce06aa6c51aca945105c075139aed7e55a2
ARROW_S2N_TLS_BUILD_VERSION=v1.3.35
ARROW_S2N_TLS_BUILD_SHA256_CHECKSUM=9d32b26e6bfcc058d98248bf8fc231537e347395dd89cf62bb432b55c5da990d
-ARROW_THRIFT_BUILD_VERSION=0.16.0
-ARROW_THRIFT_BUILD_SHA256_CHECKSUM=f460b5c1ca30d8918ff95ea3eb6291b3951cf518553566088f3f2be8981f6209
+ARROW_THRIFT_BUILD_VERSION=0.20.0
+ARROW_THRIFT_BUILD_SHA256_CHECKSUM=b5d8311a779470e1502c027f428a1db542f5c051c8e1280ccd2163fa935ff2d6
ARROW_UCX_BUILD_VERSION=1.12.1
ARROW_UCX_BUILD_SHA256_CHECKSUM=9bef31aed0e28bf1973d28d74d9ac4f8926c43ca3b7010bd22a084e164e31b71
ARROW_UTF8PROC_BUILD_VERSION=v2.7.0
@@ -164,7 +164,7 @@ DEPENDENCIES=(
"ARROW_RE2_URL re2-${ARROW_RE2_BUILD_VERSION}.tar.gz https://github.com/google/re2/archive/${ARROW_RE2_BUILD_VERSION}.tar.gz"
"ARROW_S2N_TLS_URL s2n-${ARROW_S2N_TLS_BUILD_VERSION}.tar.gz https://github.com/aws/s2n-tls/archive/${ARROW_S2N_TLS_BUILD_VERSION}.tar.gz"
"ARROW_SNAPPY_URL snappy-${ARROW_SNAPPY_BUILD_VERSION}.tar.gz https://github.com/google/snappy/archive/${ARROW_SNAPPY_BUILD_VERSION}.tar.gz"
- "ARROW_THRIFT_URL thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz https://archive.apache.org/dist/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
+ "ARROW_THRIFT_URL thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz https://dlcdn.apache.org/thrift/${ARROW_THRIFT_BUILD_VERSION}/thrift-${ARROW_THRIFT_BUILD_VERSION}.tar.gz"
"ARROW_UCX_URL ucx-${ARROW_UCX_BUILD_VERSION}.tar.gz https://github.com/openucx/ucx/archive/v${ARROW_UCX_BUILD_VERSION}.tar.gz"
"ARROW_UTF8PROC_URL utf8proc-${ARROW_UTF8PROC_BUILD_VERSION}.tar.gz https://github.com/JuliaStrings/utf8proc/archive/${ARROW_UTF8PROC_BUILD_VERSION}.tar.gz"
"ARROW_XSIMD_URL xsimd-${ARROW_XSIMD_BUILD_VERSION}.tar.gz https://github.com/xtensor-stack/xsimd/archive/${ARROW_XSIMD_BUILD_VERSION}.tar.gz"
(arrow) 05/15: GH-44453: [Release] Add shebang to `07-matlab-upload.sh` (#44454)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit c4cd5a263f38551f88572ec6e6a11172926cfe7f
Author: Sutou Kouhei <ko...@clear-code.com>
AuthorDate: Thu Oct 17 23:59:16 2024 +0900
GH-44453: [Release] Add shebang to `07-matlab-upload.sh` (#44454)
### Rationale for this change
Executable scripts should have shebang.
### What changes are included in this PR?
Add `#!/usr/bin/env bash`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #44453
Authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Jacob Wujciak-Jens <ja...@wujciak.de>
---
dev/release/07-matlab-upload.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/dev/release/07-matlab-upload.sh b/dev/release/07-matlab-upload.sh
index 803ceadb35..fa3f2f1086 100755
--- a/dev/release/07-matlab-upload.sh
+++ b/dev/release/07-matlab-upload.sh
@@ -1,3 +1,5 @@
+#!/usr/bin/env bash
+#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
(arrow) 03/15: GH-44360: [C#] Fix Flight DoExchange incompatibility with C++ implementation (#44424)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit a803dd9df691f7816239ef3813bc507d87ca5ed9
Author: Adam Reeve <ad...@gmail.com>
AuthorDate: Wed Oct 16 13:33:23 2024 +1300
GH-44360: [C#] Fix Flight DoExchange incompatibility with C++ implementation (#44424)
### Rationale for this change
See #44360
### What changes are included in this PR?
* Adds a new integration test to allow testing `do_exchange` between C++/Python and .NET.
* Updates the Flight stream reader to handle when a descriptor is sent in the first message without any schema.
### Are these changes tested?
* Yes, using the new integration test.
### Are there any user-facing changes?
No
* GitHub Issue: #44360
Authored-by: Adam Reeve <ad...@gmail.com>
Signed-off-by: Curt Hagenlocher <cu...@hagenlocher.org>
---
.../flight/integration_tests/test_integration.cc | 129 +++++++++++++++++++++
.../Internal/RecordBatchReaderImplementation.cs | 61 +++++-----
.../FlightClientCommand.cs | 18 ++-
.../FlightServerCommand.cs | 33 +++++-
.../IScenario.cs | 35 ++++++
.../Scenarios/DoExchangeEchoScenario.cs | 122 +++++++++++++++++++
.../{ => Scenarios}/JsonTestScenario.cs | 23 ++--
.../Startup.cs | 20 +---
csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs | 7 --
.../test/Apache.Arrow.Flight.Tests/FlightTests.cs | 1 -
dev/archery/archery/integration/runner.py | 6 +
docs/source/status.rst | 19 ++-
12 files changed, 385 insertions(+), 89 deletions(-)
diff --git a/cpp/src/arrow/flight/integration_tests/test_integration.cc b/cpp/src/arrow/flight/integration_tests/test_integration.cc
index da6fcf81eb..f38076822c 100644
--- a/cpp/src/arrow/flight/integration_tests/test_integration.cc
+++ b/cpp/src/arrow/flight/integration_tests/test_integration.cc
@@ -19,6 +19,7 @@
#include <iostream>
#include <memory>
+#include <numeric>
#include <string>
#include <unordered_map>
#include <utility>
@@ -1026,6 +1027,131 @@ class AppMetadataFlightInfoEndpointScenario : public Scenario {
}
};
+/// \brief The server used for testing do_exchange
+class DoExchangeServer : public FlightServerBase {
+ public:
+ DoExchangeServer() : FlightServerBase() {}
+
+ Status DoExchange(const ServerCallContext& context,
+ std::unique_ptr<FlightMessageReader> reader,
+ std::unique_ptr<FlightMessageWriter> writer) override {
+ if (reader->descriptor().type != FlightDescriptor::DescriptorType::CMD) {
+ return Status::Invalid("Must provide a command descriptor");
+ }
+
+ const std::string& cmd = reader->descriptor().cmd;
+ if (cmd == "echo") {
+ return RunEchoExchange(reader, writer);
+ } else {
+ return Status::NotImplemented("Command not implemented: ", cmd);
+ }
+ }
+
+ private:
+ static Status RunEchoExchange(std::unique_ptr<FlightMessageReader>& reader,
+ std::unique_ptr<FlightMessageWriter>& writer) {
+ FlightStreamChunk chunk;
+ bool begun = false;
+ while (true) {
+ ARROW_ASSIGN_OR_RAISE(chunk, reader->Next());
+ if (!chunk.data && !chunk.app_metadata) {
+ break;
+ }
+ if (!begun && chunk.data) {
+ begun = true;
+ RETURN_NOT_OK(writer->Begin(chunk.data->schema()));
+ }
+ if (chunk.data && chunk.app_metadata) {
+ RETURN_NOT_OK(writer->WriteWithMetadata(*chunk.data, chunk.app_metadata));
+ } else if (chunk.data) {
+ RETURN_NOT_OK(writer->WriteRecordBatch(*chunk.data));
+ } else if (chunk.app_metadata) {
+ RETURN_NOT_OK(writer->WriteMetadata(chunk.app_metadata));
+ }
+ }
+ return Status::OK();
+ }
+};
+
+/// \brief The DoExchangeEcho scenario.
+///
+/// This tests that the client and server can perform a two-way data exchange.
+///
+/// The server should echo back any data sent by the client.
+class DoExchangeEchoScenario : public Scenario {
+ Status MakeServer(std::unique_ptr<FlightServerBase>* server,
+ FlightServerOptions* options) override {
+ *server = std::make_unique<DoExchangeServer>();
+ return Status::OK();
+ }
+
+ Status MakeClient(FlightClientOptions* options) override { return Status::OK(); }
+
+ Status RunClient(std::unique_ptr<FlightClient> client) override {
+ auto descriptor = FlightDescriptor::Command("echo");
+ FlightCallOptions call_options;
+
+ ARROW_ASSIGN_OR_RAISE(auto do_exchange_result,
+ client->DoExchange(call_options, descriptor));
+ std::unique_ptr<FlightStreamWriter> writer = std::move(do_exchange_result.writer);
+ std::unique_ptr<FlightStreamReader> reader = std::move(do_exchange_result.reader);
+
+ auto schema = arrow::schema({field("x", int32(), false)});
+ ARROW_RETURN_NOT_OK(writer->Begin(schema));
+
+ ARROW_ASSIGN_OR_RAISE(auto builder,
+ RecordBatchBuilder::Make(schema, arrow::default_memory_pool()));
+
+ for (int batch_idx = 0; batch_idx < 4; ++batch_idx) {
+ auto int_builder = builder->GetFieldAs<Int32Builder>(0);
+ std::vector<int32_t> batch_data(10);
+ std::iota(batch_data.begin(), batch_data.end(), batch_idx);
+ ARROW_RETURN_NOT_OK(int_builder->AppendValues(batch_data));
+ ARROW_ASSIGN_OR_RAISE(auto record_batch, builder->Flush());
+
+ std::string app_metadata = std::to_string(batch_idx);
+ bool write_metadata = batch_idx % 2 == 0;
+
+ if (write_metadata) {
+ ARROW_RETURN_NOT_OK(
+ writer->WriteWithMetadata(*record_batch, Buffer::FromString(app_metadata)));
+ } else {
+ ARROW_RETURN_NOT_OK(writer->WriteRecordBatch(*record_batch));
+ }
+
+ ARROW_ASSIGN_OR_RAISE(auto read_result, reader->Next());
+ if (read_result.data == nullptr) {
+ return Status::Invalid("Received null data");
+ }
+ if (!read_result.data->Equals(*record_batch)) {
+ return Status::Invalid("Read data doesn't match expected data for batch ",
+ std::to_string(batch_idx), ".\n", "Expected:\n",
+ record_batch->ToString(), "Actual:\n",
+ read_result.data->ToString());
+ }
+
+ if (write_metadata) {
+ if (read_result.app_metadata == nullptr) {
+ return Status::Invalid("Received null app metadata");
+ }
+ if (read_result.app_metadata->ToString() != app_metadata) {
+ return Status::Invalid("Read metadata doesn't match expected for batch ",
+ std::to_string(batch_idx), ".\n", "Expected:\n",
+ app_metadata, "\nActual:\n",
+ read_result.app_metadata->ToString());
+ }
+ } else if (read_result.app_metadata != nullptr) {
+ return Status::Invalid("Expected no app metadata but received non-null metadata");
+ }
+ }
+
+ ARROW_RETURN_NOT_OK(writer->DoneWriting());
+ ARROW_RETURN_NOT_OK(writer->Close());
+
+ return Status::OK();
+ }
+};
+
/// \brief Schema to be returned for mocking the statement/prepared statement results.
///
/// Must be the same across all languages.
@@ -2283,6 +2409,9 @@ Status GetScenario(const std::string& scenario_name, std::shared_ptr<Scenario>*
} else if (scenario_name == "app_metadata_flight_info_endpoint") {
*out = std::make_shared<AppMetadataFlightInfoEndpointScenario>();
return Status::OK();
+ } else if (scenario_name == "do_exchange:echo") {
+ *out = std::make_shared<DoExchangeEchoScenario>();
+ return Status::OK();
} else if (scenario_name == "flight_sql") {
*out = std::make_shared<FlightSqlScenario>();
return Status::OK();
diff --git a/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs b/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs
index 99876bf769..22d0bd84fe 100644
--- a/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs
+++ b/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs
@@ -69,42 +69,43 @@ namespace Apache.Arrow.Flight.Internal
public override async ValueTask ReadSchemaAsync(CancellationToken cancellationToken)
{
- if (HasReadSchema)
+ while (!HasReadSchema)
{
- return;
- }
-
- var moveNextResult = await _flightDataStream.MoveNext(cancellationToken).ConfigureAwait(false);
-
- if (!moveNextResult)
- {
- throw new Exception("No records or schema in this flight");
- }
+ var moveNextResult = await _flightDataStream.MoveNext(cancellationToken).ConfigureAwait(false);
+ if (!moveNextResult)
+ {
+ throw new Exception("No records or schema in this flight");
+ }
- //AppMetadata will never be null, but length 0 if empty
- //Those are skipped
- if(_flightDataStream.Current.AppMetadata.Length > 0)
- {
- _applicationMetadatas.Add(_flightDataStream.Current.AppMetadata);
- }
+ if (_flightDescriptor == null && _flightDataStream.Current.FlightDescriptor != null)
+ {
+ _flightDescriptor = new FlightDescriptor(_flightDataStream.Current.FlightDescriptor);
+ }
- var header = _flightDataStream.Current.DataHeader.Memory;
- Message message = Message.GetRootAsMessage(
- ArrowReaderImplementation.CreateByteBuffer(header));
+ // AppMetadata will never be null, but length 0 if empty
+ // Those are skipped
+ if(_flightDataStream.Current.AppMetadata.Length > 0)
+ {
+ _applicationMetadatas.Add(_flightDataStream.Current.AppMetadata);
+ }
+ var header = _flightDataStream.Current.DataHeader.Memory;
+ if (header.IsEmpty)
+ {
+ // Clients may send a first message with a descriptor only and no schema
+ continue;
+ }
- if(_flightDataStream.Current.FlightDescriptor != null)
- {
- _flightDescriptor = new FlightDescriptor(_flightDataStream.Current.FlightDescriptor);
- }
+ Message message = Message.GetRootAsMessage(ArrowReaderImplementation.CreateByteBuffer(header));
- switch (message.HeaderType)
- {
- case MessageHeader.Schema:
- _schema = FlightMessageSerializer.DecodeSchema(message.ByteBuffer);
- break;
- default:
- throw new Exception($"Expected schema as the first message, but got: {message.HeaderType.ToString()}");
+ switch (message.HeaderType)
+ {
+ case MessageHeader.Schema:
+ _schema = FlightMessageSerializer.DecodeSchema(message.ByteBuffer);
+ break;
+ default:
+ throw new Exception($"Expected schema as the first message, but got: {message.HeaderType.ToString()}");
+ }
}
}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs
index d9e0ff5230..a26bcf07ec 100644
--- a/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs
@@ -16,6 +16,7 @@
using System;
using System.IO;
using System.Threading.Tasks;
+using Apache.Arrow.Flight.IntegrationTest.Scenarios;
namespace Apache.Arrow.Flight.IntegrationTest;
@@ -34,18 +35,13 @@ public class FlightClientCommand
public async Task Execute()
{
- if (!string.IsNullOrEmpty(_scenario))
+ IScenario scenario = _scenario switch
{
- // No named scenarios are currently implemented
- throw new Exception($"Scenario '{_scenario}' is not supported.");
- }
+ null => new JsonTestScenario(_jsonFileInfo),
+ "do_exchange:echo" => new DoExchangeEchoScenario(),
+ _ => throw new NotSupportedException($"Scenario '{_scenario}' is not supported"),
+ };
- if (!(_jsonFileInfo?.Exists ?? false))
- {
- throw new Exception($"Invalid JSON file path '{_jsonFileInfo?.FullName}'");
- }
-
- var scenario = new JsonTestScenario(_port, _jsonFileInfo);
- await scenario.RunClient().ConfigureAwait(false);
+ await scenario.RunClient(_port).ConfigureAwait(false);
}
}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightServerCommand.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightServerCommand.cs
index c3a7694485..38f14b7899 100644
--- a/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightServerCommand.cs
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightServerCommand.cs
@@ -16,6 +16,8 @@
using System;
using System.Net;
using System.Threading.Tasks;
+using Apache.Arrow.Flight.IntegrationTest.Scenarios;
+using Apache.Arrow.Flight.Server;
using Apache.Arrow.Flight.TestWeb;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server;
@@ -23,6 +25,8 @@ using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Logging.Console;
namespace Apache.Arrow.Flight.IntegrationTest;
@@ -37,11 +41,12 @@ public class FlightServerCommand
public async Task Execute()
{
- if (!string.IsNullOrEmpty(_scenario))
+ IScenario scenario = _scenario switch
{
- // No named scenarios are currently implemented
- throw new Exception($"Scenario '{_scenario}' is not supported.");
- }
+ null => null,
+ "do_exchange:echo" => new DoExchangeEchoScenario(),
+ _ => throw new NotSupportedException($"Scenario {_scenario} is not supported")
+ };
var host = Host.CreateDefaultBuilder()
.ConfigureWebHostDefaults(webBuilder =>
@@ -51,6 +56,26 @@ public class FlightServerCommand
{
options.Listen(IPEndPoint.Parse("127.0.0.1:0"), l => l.Protocols = HttpProtocols.Http2);
})
+ .ConfigureServices(services =>
+ {
+ if (scenario == null)
+ {
+ // Use the TestFlightServer for JSON based integration tests
+ services.AddGrpc().AddFlightServer<TestFlightServer>();
+ services.AddSingleton(new FlightStore());
+ }
+ else
+ {
+ // Use a scenario-specific server implementation
+ services.AddGrpc().Services.AddScoped<FlightServer>(_ => scenario.MakeServer());
+ }
+
+ // The integration tests rely on the port being written to the first line of stdout,
+ // so send all logging to stderr.
+ services.Configure<ConsoleLoggerOptions>(
+ o => o.LogToStandardErrorThreshold = LogLevel.Debug);
+
+ })
.UseStartup<Startup>();
})
.Build();
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/IScenario.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/IScenario.cs
new file mode 100644
index 0000000000..41ed631f33
--- /dev/null
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/IScenario.cs
@@ -0,0 +1,35 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System.Threading.Tasks;
+using Apache.Arrow.Flight.Server;
+
+namespace Apache.Arrow.Flight.IntegrationTest;
+
+/// <summary>
+/// A Flight integration test scenario
+/// </summary>
+internal interface IScenario
+{
+ /// <summary>
+ /// Create a FlightServer instance to run the scenario
+ /// </summary>
+ FlightServer MakeServer();
+
+ /// <summary>
+ /// Run the scenario using a Flight client
+ /// </summary>
+ Task RunClient(int serverPort);
+}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/DoExchangeEchoScenario.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/DoExchangeEchoScenario.cs
new file mode 100644
index 0000000000..6e9b2696bb
--- /dev/null
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/DoExchangeEchoScenario.cs
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Linq;
+using System.Threading;
+using System.Threading.Tasks;
+using Apache.Arrow.Flight.Client;
+using Apache.Arrow.Flight.Server;
+using Google.Protobuf;
+using Grpc.Core;
+using Grpc.Net.Client;
+using Grpc.Net.Client.Balancer;
+using Microsoft.Extensions.DependencyInjection;
+
+namespace Apache.Arrow.Flight.IntegrationTest.Scenarios;
+
+internal class DoExchangeServer : FlightServer
+{
+ public override async Task DoExchange(
+ FlightServerRecordBatchStreamReader requestStream,
+ FlightServerRecordBatchStreamWriter responseStream,
+ ServerCallContext context)
+ {
+ var descriptor = await requestStream.FlightDescriptor;
+ var command = descriptor.Command?.ToStringUtf8();
+ if (command != "echo")
+ {
+ throw new Exception($"Unsupported command: '{command}'");
+ }
+
+ while (await requestStream.MoveNext())
+ {
+ await responseStream.WriteAsync(
+ requestStream.Current, requestStream.ApplicationMetadata.FirstOrDefault());
+ }
+ }
+}
+
+internal class DoExchangeEchoScenario : IScenario
+{
+ public FlightServer MakeServer() => new DoExchangeServer();
+
+ public async Task RunClient(int serverPort)
+ {
+ var services = new ServiceCollection();
+ services.AddSingleton<ResolverFactory>(new GrpcTcpResolverFactory());
+ var serviceProvider = services.BuildServiceProvider();
+
+ var address = $"grpc+tcp://localhost:{serverPort}";
+ using var channel = GrpcChannel.ForAddress(
+ address,
+ new GrpcChannelOptions
+ {
+ ServiceProvider = serviceProvider,
+ Credentials = ChannelCredentials.Insecure
+ });
+
+ var client = new FlightClient(channel);
+ var descriptor = FlightDescriptor.CreateCommandDescriptor("echo");
+ using var exchange = client.DoExchange(descriptor);
+
+ using var writer = exchange.RequestStream;
+ using var reader = exchange.ResponseStream;
+
+ for (var batchIdx = 0; batchIdx < 4; batchIdx++)
+ {
+ using var batch = new RecordBatch.Builder()
+ .Append(
+ "x",
+ nullable: false,
+ array: new Int32Array.Builder().AppendRange(Enumerable.Range(batchIdx, 10)).Build())
+ .Build();
+
+ var expectedMetadata = $"{batchIdx}";
+ var writeMetadata = batchIdx % 2 == 0;
+ if (writeMetadata)
+ {
+ await writer.WriteAsync(batch, ByteString.CopyFromUtf8(expectedMetadata));
+ }
+ else
+ {
+ await writer.WriteAsync(batch);
+ }
+
+ if (!await reader.MoveNext(CancellationToken.None))
+ {
+ throw new Exception("Unexpected end of read stream");
+ }
+
+ var readMetadata = reader.ApplicationMetadata?.FirstOrDefault()?.ToStringUtf8();
+
+ if (writeMetadata && readMetadata != expectedMetadata)
+ {
+ throw new Exception($"Expected metadata '{expectedMetadata}' but received '{readMetadata}'");
+ }
+ if (!writeMetadata && readMetadata != null)
+ {
+ throw new Exception($"Unexpected metadata received: '{readMetadata}'");
+ }
+ }
+
+ await writer.CompleteAsync();
+
+ if (await reader.MoveNext(CancellationToken.None))
+ {
+ throw new Exception("Expected end of read stream");
+ }
+ }
+}
diff --git a/csharp/test/Apache.Arrow.Flight.IntegrationTest/JsonTestScenario.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs
similarity index 91%
rename from csharp/test/Apache.Arrow.Flight.IntegrationTest/JsonTestScenario.cs
rename to csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs
index f4f3ac28bf..4f7fed7435 100644
--- a/csharp/test/Apache.Arrow.Flight.IntegrationTest/JsonTestScenario.cs
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs
@@ -18,6 +18,7 @@ using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Apache.Arrow.Flight.Client;
+using Apache.Arrow.Flight.Server;
using Apache.Arrow.IntegrationTest;
using Apache.Arrow.Tests;
using Apache.Arrow.Types;
@@ -27,20 +28,23 @@ using Grpc.Core;
using Grpc.Net.Client.Balancer;
using Microsoft.Extensions.DependencyInjection;
-namespace Apache.Arrow.Flight.IntegrationTest;
+namespace Apache.Arrow.Flight.IntegrationTest.Scenarios;
/// <summary>
/// A test scenario defined using a JSON data file
/// </summary>
-internal class JsonTestScenario
+internal class JsonTestScenario : IScenario
{
- private readonly int _serverPort;
private readonly FileInfo _jsonFile;
private readonly ServiceProvider _serviceProvider;
- public JsonTestScenario(int serverPort, FileInfo jsonFile)
+ public JsonTestScenario(FileInfo jsonFile)
{
- _serverPort = serverPort;
+ if (!(jsonFile?.Exists ?? false))
+ {
+ throw new Exception($"Invalid JSON file path '{jsonFile?.FullName}'");
+ }
+
_jsonFile = jsonFile;
var services = new ServiceCollection();
@@ -48,9 +52,14 @@ internal class JsonTestScenario
_serviceProvider = services.BuildServiceProvider();
}
- public async Task RunClient()
+ public FlightServer MakeServer()
+ {
+ throw new NotImplementedException();
+ }
+
+ public async Task RunClient(int serverPort)
{
- var address = $"grpc+tcp://localhost:{_serverPort}";
+ var address = $"grpc+tcp://localhost:{serverPort}";
using var channel = GrpcChannel.ForAddress(
address,
new GrpcChannelOptions
diff --git a/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Startup.cs
similarity index 66%
copy from csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs
copy to csharp/test/Apache.Arrow.Flight.IntegrationTest/Startup.cs
index d1cfe9e445..7e29d1997e 100644
--- a/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs
+++ b/csharp/test/Apache.Arrow.Flight.IntegrationTest/Startup.cs
@@ -16,30 +16,12 @@
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
-using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
-using Microsoft.Extensions.Logging;
-using Microsoft.Extensions.Logging.Console;
-namespace Apache.Arrow.Flight.TestWeb
+namespace Apache.Arrow.Flight.IntegrationTest
{
public class Startup
{
- // This method gets called by the runtime. Use this method to add services to the container.
- // For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
- public void ConfigureServices(IServiceCollection services)
- {
- services.AddGrpc()
- .AddFlightServer<TestFlightServer>();
-
- services.AddSingleton(new FlightStore());
-
- // The integration tests rely on the port being written to the first line of stdout,
- // so send all logging to stderr.
- services.Configure<ConsoleLoggerOptions>(
- o => o.LogToStandardErrorThreshold = LogLevel.Debug);
- }
-
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
diff --git a/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs b/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs
index d1cfe9e445..68ce378ccd 100644
--- a/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs
+++ b/csharp/test/Apache.Arrow.Flight.TestWeb/Startup.cs
@@ -18,8 +18,6 @@ using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
-using Microsoft.Extensions.Logging;
-using Microsoft.Extensions.Logging.Console;
namespace Apache.Arrow.Flight.TestWeb
{
@@ -33,11 +31,6 @@ namespace Apache.Arrow.Flight.TestWeb
.AddFlightServer<TestFlightServer>();
services.AddSingleton(new FlightStore());
-
- // The integration tests rely on the port being written to the first line of stdout,
- // so send all logging to stderr.
- services.Configure<ConsoleLoggerOptions>(
- o => o.LogToStandardErrorThreshold = LogLevel.Debug);
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
diff --git a/csharp/test/Apache.Arrow.Flight.Tests/FlightTests.cs b/csharp/test/Apache.Arrow.Flight.Tests/FlightTests.cs
index 0e82673d02..350762c992 100644
--- a/csharp/test/Apache.Arrow.Flight.Tests/FlightTests.cs
+++ b/csharp/test/Apache.Arrow.Flight.Tests/FlightTests.cs
@@ -24,7 +24,6 @@ using Apache.Arrow.Tests;
using Google.Protobuf;
using Grpc.Core;
using Grpc.Core.Utils;
-using Python.Runtime;
using Xunit;
namespace Apache.Arrow.Flight.Tests
diff --git a/dev/archery/archery/integration/runner.py b/dev/archery/archery/integration/runner.py
index 378b17d75f..781b41090d 100644
--- a/dev/archery/archery/integration/runner.py
+++ b/dev/archery/archery/integration/runner.py
@@ -669,6 +669,12 @@ def run_all_tests(with_cpp=True, with_java=True, with_js=True,
"RenewFlightEndpoint are working as expected."),
skip_testers={"JS", "C#", "Rust"},
),
+ Scenario(
+ "do_exchange:echo",
+ description=("Test the do_exchange method by "
+ "echoing data back to the client."),
+ skip_testers={"Go", "Java", "JS", "Rust"},
+ ),
Scenario(
"location:reuse_connection",
description="Ensure arrow-flight-reuse-connection is accepted.",
diff --git a/docs/source/status.rst b/docs/source/status.rst
index c838604fca..5ab35f7639 100644
--- a/docs/source/status.rst
+++ b/docs/source/status.rst
@@ -208,15 +208,15 @@ Supported features in the gRPC transport:
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
| Flight RPC Feature | C++ | Java | Go | JS | C# | Rust | Julia | Swift |
+============================================+=======+=======+=======+====+=======+=======+=======+=======+
-| All RPC methods | ✓ | ✓ | ✓ | | ✓ (1) | ✓ | | |
+| All RPC methods | ✓ | ✓ | ✓ | | ✓ | ✓ | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
-| Authentication handlers | ✓ | ✓ | ✓ | | ✓ (2) | ✓ | | |
+| Authentication handlers | ✓ | ✓ | ✓ | | ✓ (1) | ✓ | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
| Call timeouts | ✓ | ✓ | ✓ | | | ✓ | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
| Call cancellation | ✓ | ✓ | ✓ | | | ✓ | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
-| Concurrent client calls (3) | ✓ | ✓ | ✓ | | ✓ | ✓ | | |
+| Concurrent client calls (2) | ✓ | ✓ | ✓ | | ✓ | ✓ | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
| Custom middleware | ✓ | ✓ | ✓ | | | ✓ | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
@@ -228,7 +228,7 @@ Supported features in the UCX transport:
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
| Flight RPC Feature | C++ | Java | Go | JS | C# | Rust | Julia | Swift |
+============================================+=======+=======+=======+====+=======+=======+=======+=======+
-| All RPC methods | ✓ (4) | | | | | | | |
+| All RPC methods | ✓ (3) | | | | | | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
| Authentication handlers | | | | | | | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
@@ -236,7 +236,7 @@ Supported features in the UCX transport:
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
| Call cancellation | | | | | | | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
-| Concurrent client calls | ✓ (5) | | | | | | | |
+| Concurrent client calls | ✓ (4) | | | | | | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
| Custom middleware | | | | | | | | |
+--------------------------------------------+-------+-------+-------+----+-------+-------+-------+-------+
@@ -245,11 +245,10 @@ Supported features in the UCX transport:
Notes:
-* \(1) No support for Handshake or DoExchange.
-* \(2) Support using AspNetCore authentication handlers.
-* \(3) Whether a single client can support multiple concurrent calls.
-* \(4) Only support for DoExchange, DoGet, DoPut, and GetFlightInfo.
-* \(5) Each concurrent call is a separate connection to the server
+* \(1) Support using AspNetCore authentication handlers.
+* \(2) Whether a single client can support multiple concurrent calls.
+* \(3) Only support for DoExchange, DoGet, DoPut, and GetFlightInfo.
+* \(4) Each concurrent call is a separate connection to the server
(unlike gRPC where concurrent calls are multiplexed over a single
connection). This will generally provide better throughput but
consumes more resources both on the server and the client.
(arrow) 06/15: GH-44451: [Release] Retry on HTTP error in Java upload (#44452)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 6040c07ceae51963f99d2a46525b6839bb6a9ad1
Author: Sutou Kouhei <ko...@clear-code.com>
AuthorDate: Fri Oct 18 00:01:22 2024 +0900
GH-44451: [Release] Retry on HTTP error in Java upload (#44452)
### Rationale for this change
`dev/release/06-java-upload.sh` sometimes failed with the following HTTP error:
```text
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy-file (default-cli) on project standalone-pom: Failed to deploy artifacts: Could not transfer artifact org.apache.arrow:arrow-compression:pom:18.0.0 from/to apache.releases.https (https://repository.apache.org/service/local/staging/deploy/maven2): transfer failed for https://repository.apache.org/service/local/staging/deploy/maven2/org/apache/arrow/arrow-compression/18.0.0/arrow-compression-18.0.0.po [...]
```
### What changes are included in this PR?
Use `-DretryFailedDeploymentCount=10` for retry on HTTP error.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #44451
Authored-by: Sutou Kouhei <ko...@clear-code.com>
Signed-off-by: Jacob Wujciak-Jens <ja...@wujciak.de>
---
dev/release/06-java-upload.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/dev/release/06-java-upload.sh b/dev/release/06-java-upload.sh
index 9d791f8ef7..d0fd851da5 100755
--- a/dev/release/06-java-upload.sh
+++ b/dev/release/06-java-upload.sh
@@ -107,8 +107,9 @@ for pom in *.pom; do
classifiers=""
args=()
args+=(deploy:deploy-file)
- args+=(-Durl=https://repository.apache.org/service/local/staging/deploy/maven2)
args+=(-DrepositoryId=apache.releases.https)
+ args+=(-DretryFailedDeploymentCount=10)
+ args+=(-Durl=https://repository.apache.org/service/local/staging/deploy/maven2)
pom="${PWD}/${pom}"
args+=(-DpomFile="${pom}")
if [ -f "${base}.jar" ]; then
@@ -139,7 +140,7 @@ for pom in *.pom; do
args+=(-Dtypes="${types}")
args+=(-Dclassifiers="${classifiers}")
pushd "${SOURCE_DIR}"
- mvn deploy:deploy-file "${args[@]}"
+ mvn "${args[@]}"
popd
done
(arrow) 07/15: GH-44459: [Release][Packaging] Do not check wheel content when verifying binary wheels (#44460)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 587af2e67605f1d739c147cf3e784297dad59709
Author: Raúl Cumplido <ra...@gmail.com>
AuthorDate: Thu Oct 17 21:56:08 2024 +0200
GH-44459: [Release][Packaging] Do not check wheel content when verifying binary wheels (#44460)
### Rationale for this change
Currently the folder used to verify the contents of the wheels is not correct when doing a release verification of the wheels and the verification fails. The script is supposed to check wheels individually and we are already verifying the contents on the wheels individually when generating the wheels.
### What changes are included in this PR?
Stop checking wheel content on release verification as the wheel content has already been validated when generating the wheels.
There is also a minor fix on the platform tags for the macOS wheels that was missed when we updated the MACOSX_DEPLOYMENT_TARGET.
### Are these changes tested?
Yes via archery.
### Are there any user-facing changes?
No
* GitHub Issue: #44459
Authored-by: Raúl Cumplido <ra...@gmail.com>
Signed-off-by: Sutou Kouhei <ko...@clear-code.com>
---
dev/release/verify-release-candidate.sh | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh
index bf1f585581..5e5182967f 100755
--- a/dev/release/verify-release-candidate.sh
+++ b/dev/release/verify-release-candidate.sh
@@ -1057,6 +1057,10 @@ test_linux_wheels() {
local python_versions="${TEST_PYTHON_VERSIONS:-3.9 3.10 3.11 3.12 3.13}"
local platform_tags="${TEST_WHEEL_PLATFORM_TAGS:-manylinux_2_17_${arch}.manylinux2014_${arch} manylinux_2_28_${arch}}"
+ if [ "${SOURCE_KIND}" != "local" ]; then
+ local wheel_content="OFF"
+ fi
+
for python in ${python_versions}; do
local pyver=${python/m}
for platform in ${platform_tags}; do
@@ -1066,7 +1070,8 @@ test_linux_wheels() {
continue
fi
pip install pyarrow-${TEST_PYARROW_VERSION:-${VERSION}}-cp${pyver/.}-cp${python/.}-${platform}.whl
- INSTALL_PYARROW=OFF ARROW_GCS=${check_gcs} ${ARROW_DIR}/ci/scripts/python_wheel_unix_test.sh ${ARROW_SOURCE_DIR}
+ CHECK_WHEEL_CONTENT=${wheel_content:-"ON"} INSTALL_PYARROW=OFF ARROW_GCS=${check_gcs} \
+ ${ARROW_DIR}/ci/scripts/python_wheel_unix_test.sh ${ARROW_SOURCE_DIR}
done
done
}
@@ -1079,11 +1084,15 @@ test_macos_wheels() {
# apple silicon processor
if [ "$(uname -m)" = "arm64" ]; then
local python_versions="3.9 3.10 3.11 3.12 3.13"
- local platform_tags="macosx_11_0_arm64"
+ local platform_tags="macosx_12_0_arm64"
local check_flight=OFF
else
local python_versions="3.9 3.10 3.11 3.12 3.13"
- local platform_tags="macosx_10_15_x86_64"
+ local platform_tags="macosx_12_0_x86_64"
+ fi
+
+ if [ "${SOURCE_KIND}" != "local" ]; then
+ local wheel_content="OFF"
fi
# verify arch-native wheels inside an arch-native conda environment
@@ -1102,7 +1111,8 @@ test_macos_wheels() {
fi
pip install pyarrow-${VERSION}-cp${pyver/.}-cp${python/.}-${platform}.whl
- INSTALL_PYARROW=OFF ARROW_FLIGHT=${check_flight} ARROW_GCS=${check_gcs} ARROW_S3=${check_s3} \
+ CHECK_WHEEL_CONTENT=${wheel_content:-"ON"} INSTALL_PYARROW=OFF ARROW_FLIGHT=${check_flight} \
+ ARROW_GCS=${check_gcs} ARROW_S3=${check_s3} \
${ARROW_DIR}/ci/scripts/python_wheel_unix_test.sh ${ARROW_SOURCE_DIR}
done
done
(arrow) 10/15: GH-44506: [Docs][C++] Fix documentation directive for ChunkLocation (#44505)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 9e7feb94b5188071330c26c4271b57a11e50625e
Author: Bryce Mecum <pe...@gmail.com>
AuthorDate: Tue Oct 22 18:36:01 2024 -0700
GH-44506: [Docs][C++] Fix documentation directive for ChunkLocation (#44505)
### Rationale for this change
Since ChunkLocation is a typedef/alias, it needs to use the `doxygentypedef` Sphinx directive rather than `doxygenstruct` in order for it to show up in the generated HTML. Otherwise we get an error.
### What changes are included in this PR?
Just docs:
- Replace `doxygenstruct` directive with `doxygentypedef`
- Remove `:members:` directive as it's not needed
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, though just docs.
* GitHub Issue: #44506
Authored-by: Bryce Mecum <pe...@gmail.com>
Signed-off-by: Bryce Mecum <pe...@gmail.com>
---
docs/source/cpp/api/array.rst | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/docs/source/cpp/api/array.rst b/docs/source/cpp/api/array.rst
index 133a432884..a87e3810c4 100644
--- a/docs/source/cpp/api/array.rst
+++ b/docs/source/cpp/api/array.rst
@@ -88,9 +88,8 @@ Chunked Arrays
:project: arrow_cpp
:members:
-.. doxygenstruct:: arrow::ChunkLocation
+.. doxygentypedef:: arrow::ChunkLocation
:project: arrow_cpp
- :members:
.. doxygenstruct:: arrow::TypedChunkLocation
:project: arrow_cpp
(arrow) 12/15: GH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on verify-release-candidate-wheels.bat (#44462)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brycemecum pushed a commit to branch maint-18.1.0
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 7e26edf033c72b0e25583daef2c271c89fc81477
Author: Raúl Cumplido <ra...@gmail.com>
AuthorDate: Fri Oct 25 16:49:04 2024 +0200
GH-44461: [Release][Packacing][Python] Set PARQUET_TEST_DATA on verify-release-candidate-wheels.bat (#44462)
### Rationale for this change
The Windows wheel verification fails due to missing `PARQUET_TEST_DATA`
### What changes are included in this PR?
Add `PARQUET_TEST_DATA` to `verify-release-candidate-wheels.bat` which is only tested on the binary verification job.
### Are these changes tested?
Via archery
### Are there any user-facing changes?
No
* GitHub Issue: #44461
Lead-authored-by: Sutou Kouhei <ko...@clear-code.com>
Co-authored-by: Raúl Cumplido <ra...@gmail.com>
Co-authored-by: Sutou Kouhei <ko...@cozmixng.org>
Signed-off-by: Raúl Cumplido <ra...@gmail.com>
---
ci/scripts/download_tz_database.sh | 4 ++--
dev/release/verify-release-candidate-wheels.bat | 2 ++
dev/tasks/verify-rc/github.win.yml | 1 +
3 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/ci/scripts/download_tz_database.sh b/ci/scripts/download_tz_database.sh
index c4095ef9c2..b74d251a43 100755
--- a/ci/scripts/download_tz_database.sh
+++ b/ci/scripts/download_tz_database.sh
@@ -20,11 +20,11 @@
set -ex
# Download database
-curl https://data.iana.org/time-zones/releases/tzdata2021e.tar.gz --output ~/Downloads/tzdata2021e.tar.gz
+curl https://data.iana.org/time-zones/releases/tzdata2024b.tar.gz --output ~/Downloads/tzdata.tar.gz
# Extract
mkdir -p ~/Downloads/tzdata
-tar --extract --file ~/Downloads/tzdata2021e.tar.gz --directory ~/Downloads/tzdata
+tar --extract --file ~/Downloads/tzdata.tar.gz --directory ~/Downloads/tzdata
# Download Windows timezone mapping
curl https://raw.githubusercontent.com/unicode-org/cldr/master/common/supplemental/windowsZones.xml --output ~/Downloads/tzdata/windowsZones.xml
diff --git a/dev/release/verify-release-candidate-wheels.bat b/dev/release/verify-release-candidate-wheels.bat
index d846b69da8..a9a4703fae 100644
--- a/dev/release/verify-release-candidate-wheels.bat
+++ b/dev/release/verify-release-candidate-wheels.bat
@@ -55,6 +55,7 @@ python arrow\dev\release\download_rc_binaries.py %ARROW_VERSION% %RC_NUMBER% ^
call deactivate
set ARROW_TEST_DATA=%cd%\arrow\testing\data
+set PARQUET_TEST_DATA=%cd%\arrow\cpp\submodules\parquet-testing\data
CALL :verify_wheel 3.9
@@ -107,6 +108,7 @@ python -c "import pyarrow.dataset" || EXIT /B 1
pip install -r arrow\python\requirements-test.txt || EXIT /B 1
set PYARROW_TEST_CYTHON=OFF
+set TZDIR=%CONDA_ENV_PATH%\share\zoneinfo
pytest %CONDA_ENV_PATH%\Lib\site-packages\pyarrow --pdb -v || EXIT /B 1
:done
diff --git a/dev/tasks/verify-rc/github.win.yml b/dev/tasks/verify-rc/github.win.yml
index 67ebb39a87..c70fc80139 100644
--- a/dev/tasks/verify-rc/github.win.yml
+++ b/dev/tasks/verify-rc/github.win.yml
@@ -29,6 +29,7 @@ jobs:
{{ key }}: {{ value }}
{% endfor %}
{% endif %}
+ timeout-minutes: 60
steps:
{{ macros.github_checkout_arrow(fetch_depth=0)|indent }}