You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "fangchenli (via GitHub)" <gi...@apache.org> on 2026/07/11 19:51:28 UTC

[PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

fangchenli opened a new pull request, #50479:
URL: https://github.com/apache/arrow/pull/50479

   ### Rationale for this change
   
   The scalar string predicate/measurement kernels return `NotImplemented` for
   `string_view`/`binary_view` input, so evaluating a string predicate (e.g. a
   `LIKE` filter) on a view array first requires casting the whole column to
   `utf8`/`binary` — a full copy. This closes that gap so the predicates run
   directly on view arrays.
   
   Part of the view-kernel effort (umbrella #44336, tracking #39634); follows
   grouper keys (#50224) and cast (#50166). Take/filter (#43010) is handled
   separately by #50164.
   
   ### What changes are included in this PR?
   
   Adds `STRING_VIEW`/`BINARY_VIEW` kernels for the input-view, fixed-output subset:
   
   - `match_substring`(`_regex`), `match_like`, `starts_with`, `ends_with`
   - `find_substring`(`_regex`), `count_substring`(`_regex`)
   - `binary_length`, `utf8_length`
   - `string_is_ascii`, `ascii_is_*`, `utf8_is_*`
   
   Implementation notes:
   
   - `MatchSubstringImpl` gains an `if constexpr` branch that iterates view elements
     via `ArrayIterator` (the packed-offset fast path is unchanged for base
     binary/string). `match_like` and the regex / ignore-case variants route through
     the same path.
   - The applicator-based kernels (`find`/`count`/`length`/classifiers) already
     handle the view layout, so they only needed registration entries. A small
     `StringOffsetType` helper maps the view types to `Int32` output (a view element
     length is int32-sized).
   
   Kernels that **emit** strings/lists (`utf8_upper`/`lower`/`trim*`,
   `replace_substring`, `split_pattern`, `utf8_slice_codeunits`, …) are out of scope
   for a follow-up, since constructing view output is the harder half.
   
   ### Are these changes tested?
   
   Yes — new `StringViewPredicates.*` cases in `scalar_string_test.cc` cover empty,
   inlined (≤12 byte), out-of-line (>12 byte), sliced, and null values for both
   `binary_view` and `utf8_view`, asserting results match the plain string/binary
   semantics.
   
   A `MatchSubstringView` / `MatchSubstringViewCast` benchmark pair quantifies
   running the predicate directly on a view array vs. the pre-PR cast-to-utf8
   workaround. Release build, ~1M rows of avg-16-byte ASCII, mean of 3 reps:
   
   | Benchmark | Time | Throughput |
   | --- | --- | --- |
   | `MatchSubstring` — utf8, direct | 28.1 ms | 564 MiB/s |
   | `MatchSubstringView` — utf8_view, direct (this PR) | 31.3 ms | 506 MiB/s |
   | `MatchSubstringViewCast` — view → cast to utf8 → match (pre-PR workaround) | 42.0 ms | 378 MiB/s |
   
   Evaluating the predicate directly on the view array is ~1.34x faster than
   casting to utf8 first, by skipping the full-column copy. Direct view evaluation
   is ~11% slower than native utf8 (inherent to the view layout's per-element
   decode) but avoids the cast entirely.
   
   ### Are there any user-facing changes?
   
   Yes — these compute functions now accept `string_view`/`binary_view` input. The
   `compute.rst` type-support tables are updated accordingly.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50479:
URL: https://github.com/apache/arrow/pull/50479#discussion_r3636649298


##########
cpp/src/arrow/compute/kernels/scalar_string_ascii.cc:
##########
@@ -1337,27 +1343,47 @@ struct RegexSubstringMatcher {
 
 template <typename Type, typename Matcher>
 struct MatchSubstringImpl {
-  using offset_type = typename Type::offset_type;
-
   static Status Exec(KernelContext* ctx, const ExecSpan& batch, ExecResult* out,
                      const Matcher* matcher) {
-    StringBoolTransform<Type>(
-        ctx, batch,
-        [&matcher](const void* raw_offsets, const uint8_t* data, int64_t length,
-                   int64_t output_offset, uint8_t* output) {
-          const offset_type* offsets = reinterpret_cast<const offset_type*>(raw_offsets);
-          FirstTimeBitmapWriter bitmap_writer(output, output_offset, length);
-          for (int64_t i = 0; i < length; ++i) {
-            const char* current_data = reinterpret_cast<const char*>(data + offsets[i]);
-            int64_t current_length = offsets[i + 1] - offsets[i];
-            if (matcher->Match(std::string_view(current_data, current_length))) {
+    if constexpr (is_binary_view_like_type<Type>::value) {
+      // Views have no packed offset buffer, so evaluate per element. Null slots are
+      // skipped: a view's null header is not validated and may carry a bogus
+      // buffer_index/offset that decoding would dereference.
+      const ArraySpan& input = batch[0].array;
+      ArraySpan* out_arr = out->array_span_mutable();
+      FirstTimeBitmapWriter bitmap_writer(out_arr->buffers[1].data, out_arr->offset,
+                                          input.length);
+      VisitArrayValuesInline<Type>(

Review Comment:
   Note: I think ideally we would also use `VisitArrayValuesInline` for other binary types, because the matcher may be expensive and it's better to skip it on null entries.
   
   (for example, if the array has 99% nulls, it would be wasteful to run a regexp matcher on each underlying empty slot)



##########
cpp/src/arrow/compute/kernels/scalar_string_benchmark.cc:
##########
@@ -77,6 +87,28 @@ static void MatchSubstring(benchmark::State& state) {
   UnaryStringBenchmark(state, "match_substring", &options);
 }
 
+// Predicate run directly on a string_view array (the path added by this PR).
+static void MatchSubstringView(benchmark::State& state) {
+  MatchSubstringOptions options("abac");
+  UnaryStringBenchmark(state, "match_substring", &options, utf8_view());
+}
+
+// Baseline: the pre-PR workaround of casting the view column to utf8 first. The
+// gap vs. MatchSubstringView is the cast that direct view support avoids.
+static void MatchSubstringViewCast(benchmark::State& state) {

Review Comment:
   We don't care about benchmarking this IMHO. It was useful to judge this PR's usefulness, but it's not useful for measuring Arrow C++ performance.



##########
cpp/src/arrow/compute/kernels/scalar_string_ascii.cc:
##########
@@ -1611,6 +1637,19 @@ const FunctionDoc match_like_doc(
     {"strings"}, "MatchSubstringOptions", /*options_required=*/true);
 #endif
 
+// Register the view kernels for a match-substring-style predicate. Registered per
+// view type (not via GenerateVarBinaryViewBase) so utf8_view -> StringViewType keeps
+// the is_utf8 distinction used by ignore_case/regex folding.

Review Comment:
   I've opened https://github.com/apache/arrow/issues/50615 as it's frankly a pity to duplicate kernel code just for the is_utf8 flag.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50479:
URL: https://github.com/apache/arrow/pull/50479#issuecomment-4948540890

   :warning: GitHub issue #50478 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #50479:
URL: https://github.com/apache/arrow/pull/50479#issuecomment-5075382057

   @pitrou do you want to take another look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "fangchenli (via GitHub)" <gi...@apache.org>.
fangchenli commented on code in PR #50479:
URL: https://github.com/apache/arrow/pull/50479#discussion_r3576059277


##########
cpp/src/arrow/compute/kernels/scalar_string_test.cc:
##########
@@ -2690,4 +2694,129 @@ TEST(TestStringKernels, UnicodeLibraryAssumptions) {
 }
 #endif
 
+// ----------------------------------------------------------------------
+// View type support for scalar string predicate/measurement kernels.
+
+// Mixes empty, inlined (<= 12 bytes), out-of-line (> 12 bytes), and null values
+// to cover the view layout.
+constexpr auto kViewInput =
+    R"(["", "cat", null, "the quick brown fox", "concatenation", "a cat sat", "CAT"])";
+
+TEST(TestStringViewPredicates, MatchSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_substring", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, true, false]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, StartsWithEndsWith) {
+  MatchSubstringOptions starts{"c"};
+  MatchSubstringOptions ends{"t"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("starts_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, false, false]", &starts);
+    CheckScalarUnary("ends_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, false, true, false]", &ends);
+  }
+}
+
+TEST(TestStringViewPredicates, FindSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("find_substring", ty, kViewInput, int32(),
+                     "[-1, 0, null, -1, 3, 2, -1]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, CountSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("count_substring", ty, kViewInput, int32(),
+                     "[0, 1, null, 0, 1, 1, 0]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, BinaryLength) {
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("binary_length", ty, kViewInput, int32(),
+                     "[0, 3, null, 19, 13, 9, 3]");
+  }
+}
+
+#ifdef ARROW_WITH_RE2
+TEST(TestStringViewPredicates, MatchSubstringRegex) {
+  MatchSubstringOptions options{"a+"};
+  const auto* input = R"(["", "cat", null, "aaa banana", "concatenation"])";
+  for (const auto& ty : {binary_view(), utf8_view()}) {

Review Comment:
   Good call. Added `TestStringViewPredicates.MatchSubstringIgnoreCase` in 8082d31: it mirrors the existing `MatchSubstringIgnoreCase` coverage with pattern `"aé("`, `ignore_case=true`, and input `["abc", "aEb", "baÉ(", "aé(", "ae(", "Aé("]` → `[false, false, true, true, false, true]`. The uppercase `É` matching lowercase `é` only holds because `utf8_view` folds over the full Unicode range; if it were dispatched through the generic `BinaryViewType` path (ASCII/Latin1 folding only) that slot would come back `false`, so the test fails on that regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #50479:
URL: https://github.com/apache/arrow/pull/50479#discussion_r3574150382


##########
cpp/src/arrow/compute/kernels/scalar_string_test.cc:
##########
@@ -2690,4 +2694,129 @@ TEST(TestStringKernels, UnicodeLibraryAssumptions) {
 }
 #endif
 
+// ----------------------------------------------------------------------
+// View type support for scalar string predicate/measurement kernels.
+
+// Mixes empty, inlined (<= 12 bytes), out-of-line (> 12 bytes), and null values
+// to cover the view layout.
+constexpr auto kViewInput =
+    R"(["", "cat", null, "the quick brown fox", "concatenation", "a cat sat", "CAT"])";
+
+TEST(TestStringViewPredicates, MatchSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_substring", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, true, false]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, StartsWithEndsWith) {
+  MatchSubstringOptions starts{"c"};
+  MatchSubstringOptions ends{"t"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("starts_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, false, false]", &starts);
+    CheckScalarUnary("ends_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, false, true, false]", &ends);
+  }
+}
+
+TEST(TestStringViewPredicates, FindSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("find_substring", ty, kViewInput, int32(),
+                     "[-1, 0, null, -1, 3, 2, -1]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, CountSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("count_substring", ty, kViewInput, int32(),
+                     "[0, 1, null, 0, 1, 1, 0]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, BinaryLength) {
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("binary_length", ty, kViewInput, int32(),
+                     "[0, 3, null, 19, 13, 9, 3]");
+  }
+}
+
+#ifdef ARROW_WITH_RE2
+TEST(TestStringViewPredicates, MatchSubstringRegex) {
+  MatchSubstringOptions options{"a+"};
+  const auto* input = R"(["", "cat", null, "aaa banana", "concatenation"])";
+  for (const auto& ty : {binary_view(), utf8_view()}) {

Review Comment:
   Could we add one `utf8_view` case with `ignore_case=true` and non-ASCII input here (for example mirroring the existing `MatchSubstringIgnoreCase` coverage)?
   
   The implementation intentionally registers `utf8_view` as `StringViewType` instead of using the generic `BinaryViewType` dispatch, so it would be good to have a test that would fail if we accidentally lost the UTF-8/Latin1 distinction. The current view-specific tests are mostly ASCII, so they wouldn't catch that regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50479:
URL: https://github.com/apache/arrow/pull/50479#discussion_r3912102846


##########
cpp/src/arrow/compute/kernels/scalar_string_ascii.cc:
##########
@@ -1645,6 +1646,7 @@ void AddAsciiStringMatchSubstring(FunctionRegistry* registry) {
       DCHECK_OK(
           func->AddKernel({ty}, boolean(), std::move(exec), MatchSubstringState::Init));
     }
+    AddMatchSubstringViewKernels<MatchSubstring, PlainSubstringMatcher>(func.get());

Review Comment:
   Can we perhaps add a `GenerateTypeAgnosticVarBinaryViewBase` so that we can write:
   ```c++
       for (const auto& ty : BinaryViewTypes()) {
         auto exec =
             GenerateTypeAgnosticVarBinaryViewBase<MatchSubstring, PlainSubstringMatcher>(ty);
         DCHECK_OK(
             func->AddKernel({ty}, boolean(), std::move(exec), MatchSubstringState::Init));
       }
   ```



##########
cpp/src/arrow/compute/kernels/scalar_string_test.cc:
##########
@@ -2837,4 +2841,146 @@ TEST(TestStringKernels, UnicodeLibraryAssumptions) {
 }
 #endif
 
+// ----------------------------------------------------------------------
+// View type support for scalar string predicate/measurement kernels.
+
+// Mixes empty, inlined (<= 12 bytes), out-of-line (> 12 bytes), and null values
+// to cover the view layout.
+constexpr auto kViewInput =
+    R"(["", "cat", null, "the quick brown fox", "concatenation", "a cat sat", "CAT"])";
+
+TEST(TestStringViewPredicates, MatchSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_substring", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, true, false]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, StartsWithEndsWith) {
+  MatchSubstringOptions starts{"c"};
+  MatchSubstringOptions ends{"t"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("starts_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, false, false]", &starts);
+    CheckScalarUnary("ends_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, false, true, false]", &ends);
+  }
+}
+
+TEST(TestStringViewPredicates, FindSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("find_substring", ty, kViewInput, int32(),
+                     "[-1, 0, null, -1, 3, 2, -1]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, CountSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("count_substring", ty, kViewInput, int32(),
+                     "[0, 1, null, 0, 1, 1, 0]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, BinaryLength) {
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("binary_length", ty, kViewInput, int32(),
+                     "[0, 3, null, 19, 13, 9, 3]");
+  }
+}
+
+#ifdef ARROW_WITH_RE2
+TEST(TestStringViewPredicates, MatchSubstringRegex) {
+  MatchSubstringOptions options{"a+"};
+  const auto* input = R"(["", "cat", null, "aaa banana", "concatenation"])";
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_substring_regex", ty, input, boolean(),
+                     "[false, true, null, true, true]", &options);
+    CheckScalarUnary("find_substring_regex", ty, input, int32(), "[-1, 1, null, 0, 4]",
+                     &options);
+    CheckScalarUnary("count_substring_regex", ty, input, int32(), "[0, 1, null, 4, 2]",
+                     &options);
+  }
+}
+
+TEST(TestStringViewPredicates, MatchLike) {
+  MatchSubstringOptions contains{"%cat%"};
+  MatchSubstringOptions prefix{"c%"};
+  const auto* input = R"(["cat", "concatenation", null, "dog", "a cat sat"])";
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_like", ty, input, boolean(),
+                     "[true, true, null, false, true]", &contains);
+    CheckScalarUnary("match_like", ty, input, boolean(),
+                     "[true, true, null, false, false]", &prefix);
+  }
+}
+
+// utf8_view registers as StringViewType, so ignore_case folds the full Unicode
+// range (É matches é), not just ASCII. This would fail if utf8_view were
+// dispatched through the generic BinaryViewType path and lost that distinction.
+TEST(TestStringViewPredicates, MatchSubstringIgnoreCase) {
+  MatchSubstringOptions options{"aé(", /*ignore_case=*/true};
+  CheckScalarUnary("match_substring", utf8_view(),

Review Comment:
   Can we add the same test for BinaryView to check and showcase the different results?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "fangchenli (via GitHub)" <gi...@apache.org>.
fangchenli commented on code in PR #50479:
URL: https://github.com/apache/arrow/pull/50479#discussion_r3648182845


##########
cpp/src/arrow/compute/kernels/scalar_string_test.cc:
##########
@@ -2690,4 +2694,139 @@ TEST(TestStringKernels, UnicodeLibraryAssumptions) {
 }
 #endif
 
+// ----------------------------------------------------------------------
+// View type support for scalar string predicate/measurement kernels.
+
+// Mixes empty, inlined (<= 12 bytes), out-of-line (> 12 bytes), and null values
+// to cover the view layout.
+constexpr auto kViewInput =
+    R"(["", "cat", null, "the quick brown fox", "concatenation", "a cat sat", "CAT"])";
+
+TEST(TestStringViewPredicates, MatchSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_substring", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, true, false]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, StartsWithEndsWith) {
+  MatchSubstringOptions starts{"c"};
+  MatchSubstringOptions ends{"t"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("starts_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, false, false]", &starts);
+    CheckScalarUnary("ends_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, false, true, false]", &ends);
+  }
+}
+
+TEST(TestStringViewPredicates, FindSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("find_substring", ty, kViewInput, int32(),
+                     "[-1, 0, null, -1, 3, 2, -1]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, CountSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("count_substring", ty, kViewInput, int32(),
+                     "[0, 1, null, 0, 1, 1, 0]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, BinaryLength) {
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("binary_length", ty, kViewInput, int32(),
+                     "[0, 3, null, 19, 13, 9, 3]");
+  }
+}
+
+#ifdef ARROW_WITH_RE2
+TEST(TestStringViewPredicates, MatchSubstringRegex) {
+  MatchSubstringOptions options{"a+"};
+  const auto* input = R"(["", "cat", null, "aaa banana", "concatenation"])";
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_substring_regex", ty, input, boolean(),
+                     "[false, true, null, true, true]", &options);
+    CheckScalarUnary("find_substring_regex", ty, input, int32(), "[-1, 1, null, 0, 4]",
+                     &options);
+    CheckScalarUnary("count_substring_regex", ty, input, int32(), "[0, 1, null, 4, 2]",
+                     &options);
+  }
+}
+
+TEST(TestStringViewPredicates, MatchLike) {
+  MatchSubstringOptions contains{"%cat%"};
+  MatchSubstringOptions prefix{"c%"};
+  const auto* input = R"(["cat", "concatenation", null, "dog", "a cat sat"])";
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_like", ty, input, boolean(),
+                     "[true, true, null, false, true]", &contains);
+    CheckScalarUnary("match_like", ty, input, boolean(),
+                     "[true, true, null, false, false]", &prefix);
+  }
+}
+
+// utf8_view registers as StringViewType, so ignore_case folds the full Unicode
+// range (É matches é), not just ASCII. This would fail if utf8_view were
+// dispatched through the generic BinaryViewType path and lost that distinction.
+TEST(TestStringViewPredicates, MatchSubstringIgnoreCase) {
+  MatchSubstringOptions options{"aé(", /*ignore_case=*/true};
+  CheckScalarUnary("match_substring", utf8_view(),
+                   R"(["abc", "aEb", "baÉ(", "aé(", "ae(", "Aé("])", boolean(),
+                   "[false, false, true, true, false, true]", &options);
+}
+#endif
+
+TEST(TestStringViewPredicates, Utf8Length) {
+  CheckScalarUnary("utf8_length", utf8_view(),
+                   R"(["", "café", null, "a long ascii string here", "ÀÉÎ"])", int32(),
+                   "[0, 4, null, 24, 3]");
+}
+
+TEST(TestStringViewPredicates, StringIsAscii) {
+  CheckScalarUnary("string_is_ascii", utf8_view(),
+                   R"(["", "café", null, "a long ascii string here", "ÀÉÎ"])", boolean(),
+                   "[true, false, null, true, false]");
+}
+
+TEST(TestStringViewPredicates, AsciiClassification) {
+  // Exercises the shared AddUnaryStringPredicate view kernel via the ascii path.
+  CheckScalarUnary("ascii_is_alpha", utf8_view(), R"(["cat", "cat123", "", null, "CAT"])",
+                   boolean(), "[true, false, false, null, true]");
+}
+
+#ifdef ARROW_WITH_UTF8PROC
+TEST(TestStringViewPredicates, Utf8Classification) {
+  // Exercises the shared AddUnaryStringPredicate view kernel via the utf8 path.
+  CheckScalarUnary("utf8_is_alpha", utf8_view(), R"(["cat", "café", "cat123", "", null])",
+                   boolean(), "[true, true, false, false, null]");
+}
+#endif  // ARROW_WITH_UTF8PROC
+
+// A validation-passing view array may hold arbitrary bytes in a null slot's
+// header, because ValidateBinaryView skips null slots. The view kernels must not
+// decode those, or they would dereference a bogus buffer_index/offset. Craft
+// ["ok", <null slot with a 0xFF header>] and confirm the kernels ignore the null.
+TEST(TestStringViewPredicates, NullSlotWithUnvalidatedHeader) {
+  auto good = ArrayFromJSON(utf8_view(), R"(["ok", null])");
+  std::vector<BinaryViewType::c_type> views(2);
+  views[0] = util::ToInlineBinaryView("ok");
+  std::memset(&views[1], 0xFF, sizeof(BinaryViewType::c_type));
+  auto data = std::make_shared<ArrayData>(*good->data());
+  data->buffers[1] = Buffer::FromVector(std::move(views));
+  auto arr = MakeArray(data);
+  ASSERT_OK(arr->ValidateFull());  // still valid: the null slot is not checked
+
+  auto expected = ArrayFromJSON(boolean(), "[true, null]");
+  MatchSubstringOptions contains{"o"};
+  ASSERT_OK_AND_ASSIGN(Datum matched, CallFunction("match_substring", {arr}, &contains));
+  AssertDatumsEqual(Datum(expected), matched);
+  ASSERT_OK_AND_ASSIGN(Datum ascii, CallFunction("string_is_ascii", {arr}));
+  AssertDatumsEqual(Datum(expected), ascii);

Review Comment:
   Done in 26453db. Extended `NullSlotWithUnvalidatedHeader` to also call `find_substring` (→ `[0, null]`) and `utf8_length` (→ `[2, null]`) on the same crafted `["ok", <0xFF null header>]` array, so the null-slot header invariant is now asserted on the applicator (`ScalarUnaryNotNull`) path as well as the two boolean-output paths.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50479:
URL: https://github.com/apache/arrow/pull/50479#discussion_r3566571472


##########
cpp/src/arrow/compute/kernels/scalar_string_ascii.cc:
##########
@@ -1337,27 +1343,45 @@ struct RegexSubstringMatcher {
 
 template <typename Type, typename Matcher>
 struct MatchSubstringImpl {
-  using offset_type = typename Type::offset_type;
-
   static Status Exec(KernelContext* ctx, const ExecSpan& batch, ExecResult* out,
                      const Matcher* matcher) {
-    StringBoolTransform<Type>(
-        ctx, batch,
-        [&matcher](const void* raw_offsets, const uint8_t* data, int64_t length,
-                   int64_t output_offset, uint8_t* output) {
-          const offset_type* offsets = reinterpret_cast<const offset_type*>(raw_offsets);
-          FirstTimeBitmapWriter bitmap_writer(output, output_offset, length);
-          for (int64_t i = 0; i < length; ++i) {
-            const char* current_data = reinterpret_cast<const char*>(data + offsets[i]);
-            int64_t current_length = offsets[i + 1] - offsets[i];
-            if (matcher->Match(std::string_view(current_data, current_length))) {
-              bitmap_writer.Set();
+    if constexpr (is_binary_view_like_type<Type>::value) {
+      // Views have no packed offset buffer, so iterate per element. Null slots
+      // are matched too (masked by output validity); reading their zero-filled
+      // view header is safe, as StringPredicateFunctor also assumes.
+      const ArraySpan& input = batch[0].array;

Review Comment:
   The view-type fast path evaluates the matcher on null slots (relying on the view header being zero-filled). For BinaryView/StringView, null slot bytes are not validated and may contain arbitrary buffer_index/offset, so calling util::FromBinaryView via ArrayIterator can dereference out-of-bounds buffers and crash even though the output validity will mask the result. This path should avoid reading view payloads for nulls.



##########
cpp/src/arrow/compute/kernels/scalar_string_internal.h:
##########
@@ -240,6 +240,8 @@ void AddUnaryStringPredicate(std::string name, FunctionRegistry* registry,
     auto exec = GenerateVarBinaryToVarBinary<StringPredicateFunctor, Predicate>(ty);
     ARROW_DCHECK_OK(func->AddKernel({ty}, boolean(), std::move(exec)));
   }
+  ARROW_DCHECK_OK(func->AddKernel(

Review Comment:
   Registering utf8_view with StringPredicateFunctor makes the predicate kernels evaluate Predicate::Call on null view slots. ArrayIterator for view types calls util::FromBinaryView unconditionally; null slot bytes are not validated (ValidateBinaryView skips nulls), so a null slot can carry an arbitrary buffer_index/offset and trigger out-of-bounds access/crash even though output validity masks the value bit. The functor should skip null slots for view inputs (or use VisitBitBlocks/VisitArrayValuesInline) before calling FromBinaryView/Predicate::Call.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-50478: [C++][Compute] Support string_view/binary_view in scalar string predicate kernels [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #50479:
URL: https://github.com/apache/arrow/pull/50479#discussion_r3581444480


##########
cpp/src/arrow/compute/kernels/scalar_string_test.cc:
##########
@@ -2690,4 +2694,139 @@ TEST(TestStringKernels, UnicodeLibraryAssumptions) {
 }
 #endif
 
+// ----------------------------------------------------------------------
+// View type support for scalar string predicate/measurement kernels.
+
+// Mixes empty, inlined (<= 12 bytes), out-of-line (> 12 bytes), and null values
+// to cover the view layout.
+constexpr auto kViewInput =
+    R"(["", "cat", null, "the quick brown fox", "concatenation", "a cat sat", "CAT"])";
+
+TEST(TestStringViewPredicates, MatchSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_substring", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, true, false]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, StartsWithEndsWith) {
+  MatchSubstringOptions starts{"c"};
+  MatchSubstringOptions ends{"t"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("starts_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, true, false, false]", &starts);
+    CheckScalarUnary("ends_with", ty, kViewInput, boolean(),
+                     "[false, true, null, false, false, true, false]", &ends);
+  }
+}
+
+TEST(TestStringViewPredicates, FindSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("find_substring", ty, kViewInput, int32(),
+                     "[-1, 0, null, -1, 3, 2, -1]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, CountSubstring) {
+  MatchSubstringOptions options{"cat"};
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("count_substring", ty, kViewInput, int32(),
+                     "[0, 1, null, 0, 1, 1, 0]", &options);
+  }
+}
+
+TEST(TestStringViewPredicates, BinaryLength) {
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("binary_length", ty, kViewInput, int32(),
+                     "[0, 3, null, 19, 13, 9, 3]");
+  }
+}
+
+#ifdef ARROW_WITH_RE2
+TEST(TestStringViewPredicates, MatchSubstringRegex) {
+  MatchSubstringOptions options{"a+"};
+  const auto* input = R"(["", "cat", null, "aaa banana", "concatenation"])";
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_substring_regex", ty, input, boolean(),
+                     "[false, true, null, true, true]", &options);
+    CheckScalarUnary("find_substring_regex", ty, input, int32(), "[-1, 1, null, 0, 4]",
+                     &options);
+    CheckScalarUnary("count_substring_regex", ty, input, int32(), "[0, 1, null, 4, 2]",
+                     &options);
+  }
+}
+
+TEST(TestStringViewPredicates, MatchLike) {
+  MatchSubstringOptions contains{"%cat%"};
+  MatchSubstringOptions prefix{"c%"};
+  const auto* input = R"(["cat", "concatenation", null, "dog", "a cat sat"])";
+  for (const auto& ty : {binary_view(), utf8_view()}) {
+    CheckScalarUnary("match_like", ty, input, boolean(),
+                     "[true, true, null, false, true]", &contains);
+    CheckScalarUnary("match_like", ty, input, boolean(),
+                     "[true, true, null, false, false]", &prefix);
+  }
+}
+
+// utf8_view registers as StringViewType, so ignore_case folds the full Unicode
+// range (É matches é), not just ASCII. This would fail if utf8_view were
+// dispatched through the generic BinaryViewType path and lost that distinction.
+TEST(TestStringViewPredicates, MatchSubstringIgnoreCase) {
+  MatchSubstringOptions options{"aé(", /*ignore_case=*/true};
+  CheckScalarUnary("match_substring", utf8_view(),
+                   R"(["abc", "aEb", "baÉ(", "aé(", "ae(", "Aé("])", boolean(),
+                   "[false, false, true, true, false, true]", &options);
+}
+#endif
+
+TEST(TestStringViewPredicates, Utf8Length) {
+  CheckScalarUnary("utf8_length", utf8_view(),
+                   R"(["", "café", null, "a long ascii string here", "ÀÉÎ"])", int32(),
+                   "[0, 4, null, 24, 3]");
+}
+
+TEST(TestStringViewPredicates, StringIsAscii) {
+  CheckScalarUnary("string_is_ascii", utf8_view(),
+                   R"(["", "café", null, "a long ascii string here", "ÀÉÎ"])", boolean(),
+                   "[true, false, null, true, false]");
+}
+
+TEST(TestStringViewPredicates, AsciiClassification) {
+  // Exercises the shared AddUnaryStringPredicate view kernel via the ascii path.
+  CheckScalarUnary("ascii_is_alpha", utf8_view(), R"(["cat", "cat123", "", null, "CAT"])",
+                   boolean(), "[true, false, false, null, true]");
+}
+
+#ifdef ARROW_WITH_UTF8PROC
+TEST(TestStringViewPredicates, Utf8Classification) {
+  // Exercises the shared AddUnaryStringPredicate view kernel via the utf8 path.
+  CheckScalarUnary("utf8_is_alpha", utf8_view(), R"(["cat", "café", "cat123", "", null])",
+                   boolean(), "[true, true, false, false, null]");
+}
+#endif  // ARROW_WITH_UTF8PROC
+
+// A validation-passing view array may hold arbitrary bytes in a null slot's
+// header, because ValidateBinaryView skips null slots. The view kernels must not
+// decode those, or they would dereference a bogus buffer_index/offset. Craft
+// ["ok", <null slot with a 0xFF header>] and confirm the kernels ignore the null.
+TEST(TestStringViewPredicates, NullSlotWithUnvalidatedHeader) {
+  auto good = ArrayFromJSON(utf8_view(), R"(["ok", null])");
+  std::vector<BinaryViewType::c_type> views(2);
+  views[0] = util::ToInlineBinaryView("ok");
+  std::memset(&views[1], 0xFF, sizeof(BinaryViewType::c_type));
+  auto data = std::make_shared<ArrayData>(*good->data());
+  data->buffers[1] = Buffer::FromVector(std::move(views));
+  auto arr = MakeArray(data);
+  ASSERT_OK(arr->ValidateFull());  // still valid: the null slot is not checked
+
+  auto expected = ArrayFromJSON(boolean(), "[true, null]");
+  MatchSubstringOptions contains{"o"};
+  ASSERT_OK_AND_ASSIGN(Datum matched, CallFunction("match_substring", {arr}, &contains));
+  AssertDatumsEqual(Datum(expected), matched);
+  ASSERT_OK_AND_ASSIGN(Datum ascii, CallFunction("string_is_ascii", {arr}));
+  AssertDatumsEqual(Datum(expected), ascii);

Review Comment:
   Optional test coverage nit: could we also exercise one fixed-width numeric-output kernel in `NullSlotWithUnvalidatedHeader`, e.g. `find_substring` or `utf8_length`? The current regression covers the two boolean-output paths, while several newly registered view kernels go through the applicator path. It looks null-safe too, but this would make the null-slot header invariant explicit across both paths.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org