You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/04/11 17:11:15 UTC
[GitHub] [arrow] vvellanki commented on a diff in pull request #12716: ARROW-16031: [C++][Gandiva] Fix Soundex errors generate
vvellanki commented on code in PR #12716:
URL: https://github.com/apache/arrow/pull/12716#discussion_r847539724
##########
cpp/src/gandiva/precompiled/string_ops.cc:
##########
@@ -2815,14 +2815,17 @@ const char* soundex_utf8(gdv_int64 ctx, const char* in, gdv_int32 in_len,
}
// The soundex code is composed by one letter and three numbers
+ char* soundex = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(ctx, in_len));
char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(ctx, 4));
- if (ret == nullptr) {
+
+ if (soundex == nullptr || ret == nullptr) {
Review Comment:
This function is returning an error on memory allocation failure. concat_ws should do the same
##########
cpp/src/gandiva/precompiled/string_ops.cc:
##########
@@ -2814,46 +2845,51 @@ const char* soundex_utf8(gdv_int64 ctx, const char* in, gdv_int32 in_len,
return "";
}
+ char* in_clean;
+ int len_clean;
+
+ // Removing numbers, spaces and special characters, and put upper all letters
+ in_clean = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(ctx, in_len));
+ memcpy(in_clean, in, in_len);
Review Comment:
We should avoid the memcpy() - in the common case when this function is used, there are no special characters..
##########
cpp/src/gandiva/precompiled/string_ops_test.cc:
##########
@@ -2310,12 +2310,43 @@ TEST(TestStringOps, TestFromHex) {
::testing::HasSubstr("Error parsing hex string, one or more bytes are not valid."));
ctx.Reset();
}
+
TEST(TestStringOps, TestSoundex) {
gandiva::ExecutionContext ctx;
auto ctx_ptr = reinterpret_cast<int64_t>(&ctx);
int32_t out_len = 0;
const char* out;
+ out = soundex_utf8(ctx_ptr, "Robert", 6, &out_len);
Review Comment:
Please add a test case where the string starts with lower case alphabets
##########
cpp/src/gandiva/precompiled/string_ops.cc:
##########
@@ -2837,21 +2840,30 @@ const char* soundex_utf8(gdv_int64 ctx, const char* in, gdv_int32 in_len,
for (int i = start_idx, l = in_len; i < l; i++) {
if (isalpha(in[i]) > 0) {
c = toupper(in[i]) - 65;
- if (mappings[c] != '0') {
- if (mappings[c] != ret[si - 1]) {
- ret[si] = mappings[c];
- si++;
- }
-
- if (si > 3) break;
+ if (mappings[c] != soundex[si - 1]) {
+ soundex[si] = mappings[c];
+ si++;
}
}
}
- if (si <= 3) {
- while (si <= 3) {
- ret[si] = '0';
- si++;
+ for (int i = 1; i < si; i++) {
+ // If the saved letter's digit is the same as the resulting first digit, remove the
+ // digit.
+ if (i == 1 && soundex[i] == mappings[ret[0] - 65]) {
+ i++;
+ }
+ if (soundex[i] != '0') {
Review Comment:
I see this skipping all vowels and anything that maps to '0'. This is step 1 in the algorithm in line 2792.
Are we implementing step 3 as well?
3. If two or more letters with the same number are adjacent in the original name
// (before step 1), only retain the first letter;
I can see this implemented in line 2843. However, I dont see the below implemented. If we are not implementing the below, please remove this from the comments
also two letters with the same number
// separated by 'h' or 'w' are coded as a single number, whereas such letters separated
// by a vowel are coded twice. This rule also applies to the first letter.
##########
cpp/src/gandiva/precompiled/string_ops.cc:
##########
@@ -2837,21 +2840,30 @@ const char* soundex_utf8(gdv_int64 ctx, const char* in, gdv_int32 in_len,
for (int i = start_idx, l = in_len; i < l; i++) {
if (isalpha(in[i]) > 0) {
c = toupper(in[i]) - 65;
- if (mappings[c] != '0') {
- if (mappings[c] != ret[si - 1]) {
- ret[si] = mappings[c];
- si++;
- }
-
- if (si > 3) break;
+ if (mappings[c] != soundex[si - 1]) {
Review Comment:
soundex[0] is not initialised and this is comparing against soundex[0]. Please initialise soundex[0] to an invalid mapping
##########
cpp/src/gandiva/precompiled/string_ops.cc:
##########
@@ -2837,21 +2840,30 @@ const char* soundex_utf8(gdv_int64 ctx, const char* in, gdv_int32 in_len,
for (int i = start_idx, l = in_len; i < l; i++) {
Review Comment:
What is need for the temporary variable l
##########
cpp/src/gandiva/precompiled/string_ops.cc:
##########
@@ -2837,21 +2840,30 @@ const char* soundex_utf8(gdv_int64 ctx, const char* in, gdv_int32 in_len,
for (int i = start_idx, l = in_len; i < l; i++) {
if (isalpha(in[i]) > 0) {
c = toupper(in[i]) - 65;
- if (mappings[c] != '0') {
- if (mappings[c] != ret[si - 1]) {
- ret[si] = mappings[c];
- si++;
- }
-
- if (si > 3) break;
+ if (mappings[c] != soundex[si - 1]) {
+ soundex[si] = mappings[c];
+ si++;
}
}
}
- if (si <= 3) {
- while (si <= 3) {
- ret[si] = '0';
- si++;
+ for (int i = 1; i < si; i++) {
+ // If the saved letter's digit is the same as the resulting first digit, remove the
+ // digit.
+ if (i == 1 && soundex[i] == mappings[ret[0] - 65]) {
Review Comment:
i == 1 is only true once. We shouldn't be checking for i==1 in the loop.
Please change this to be:
if (soundex[1] == mappings[ret[0] - 65]) {
i = 2;
} else {
i = 1;
}
before the for loop in line 2850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org