You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/03/30 01:05:56 UTC

[GitHub] [arrow] jpedroantunes opened a new pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, ‘UTF8’, replacement char) function

jpedroantunes opened a new pull request #9844:
URL: https://github.com/apache/arrow/pull/9844


   Implement CONVERT_FROM(expression, ‘UTF8’, replacement char)
   
   Converts the byte data in expression to UTF-8. Expression can be a literal string or a field name. Will replace any invalid UTF-8 characters with the replacement character.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605181037



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1258,52 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,

Review comment:
       Sure, I will correct it for this function and then create a "story" to correct it on JIRA so we can do it soon
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604220433



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;

Review comment:
       Actually we are going to consider a single char to replace.

##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));

Review comment:
       Actually we are going to consider a single char to replace. So the input length will necessarily be equals to the output length!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605123377



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -156,6 +156,17 @@ void set_error_for_invalid_utf(int64_t execution_context, char val) {
   free(error);
 }
 
+gdv_int32 validate_utf8_following_bytes(const char* data, gdv_int32 data_len,

Review comment:
       why not use bool for return type




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604052248



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  std::string str;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += 1) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0 || text_index + char_len > text_len) {
+      str += char_to_replace;

Review comment:
       char to_replace is not necessarily null-terminated




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604391508



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,39 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length

Review comment:
       Added!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605122459



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -156,6 +156,17 @@ void set_error_for_invalid_utf(int64_t execution_context, char val) {
   free(error);
 }
 
+gdv_int32 validate_utf8_following_bytes(const char* data, gdv_int32 data_len,
+                                        gdv_int32 char_index, char* invalid_char) {
+  for (int j = 1; j < data_len; ++j) {
+    if ((data[char_index + j] & 0xC0) != 0x80) {  // bytes following head-byte of glyph
+      *invalid_char = data[char_index + j];

Review comment:
       why is this invalid_char (i think you meant invalid_byte) is needed? You are doing a unnecessary allocation for this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605126448



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -156,6 +156,17 @@ void set_error_for_invalid_utf(int64_t execution_context, char val) {
   free(error);
 }
 
+gdv_int32 validate_utf8_following_bytes(const char* data, gdv_int32 data_len,

Review comment:
       Please add a FORCE_INLINE marker




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605187325



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -156,6 +156,17 @@ void set_error_for_invalid_utf(int64_t execution_context, char val) {
   free(error);
 }
 
+gdv_int32 validate_utf8_following_bytes(const char* data, gdv_int32 data_len,

Review comment:
       Corrected




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604221525



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));

Review comment:
       As defined by email, actually we are going to consider a single char to replace. So the input length will necessarily be equals to the output length!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605215739



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,44 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 char_to_replace_len, gdv_int32* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);

Review comment:
       I think I got your point of optimization. I changed for only copying to memory when detecting an invalid char, otherwise I just keep track of the bytes to be copied later.
   
   If I'm still missing something, please correct me again. I need to learn it :)

##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1258,52 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 char_to_replace_len, gdv_int32* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0) {
+      memcpy(ret + out_byte_counter, char_to_replace, 1);
+      out_byte_counter += 1;
+      // define char_len = 1 to increase text_index by 1 (as ASCII char fits in 1 byte)
+      char_len = 1;
+      continue;
+    }
+    // if the char length is greater than 0, execute another validation on MSBs
+    char* invalid_char = reinterpret_cast<char*>(malloc(char_len));
+    if (text_index + char_len > text_len || validate_utf8_following_bytes(
+        text_in, char_len, text_index, invalid_char) == 0) {
+      memcpy(ret + out_byte_counter, char_to_replace, char_len);
+    } else {
+      memcpy(ret + out_byte_counter, text_in + text_index, char_len);

Review comment:
       Corrected!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r608629370



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1257,54 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(int64_t context, const char* text_in,
+                                                    int32_t text_len,
+                                                    const char* char_to_replace,
+                                                    int32_t char_to_replace_len,
+                                                    int32_t* out_len) {
+  if (char_to_replace_len > 1) {

Review comment:
       char_to_replace can be empty string in which case return the original string

##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1257,54 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(int64_t context, const char* text_in,
+                                                    int32_t text_len,
+                                                    const char* char_to_replace,
+                                                    int32_t char_to_replace_len,
+                                                    int32_t* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte

Review comment:
       nit: better to rephrase as only supports ascii character.

##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1257,54 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(int64_t context, const char* text_in,
+                                                    int32_t text_len,
+                                                    const char* char_to_replace,
+                                                    int32_t char_to_replace_len,
+                                                    int32_t* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  int32_t valid_bytes_to_cpy = 0;
+  int32_t out_byte_counter = 0;
+  int32_t char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    // only memory copy the bytes when detect invalid char
+    if (char_len == 0 || text_index + char_len > text_len ||
+        !validate_utf8_following_bytes(text_in, char_len, text_index)) {
+      // define char_len = 1 to increase text_index by 1 (as ASCII char fits in 1 byte)
+      char_len = 1;
+      // first copy the valid bytes until now and then replace the invalid character
+      memcpy(ret + out_byte_counter, text_in + out_byte_counter, valid_bytes_to_cpy);
+      memcpy(ret + out_byte_counter + valid_bytes_to_cpy, char_to_replace, char_len);
+      out_byte_counter += valid_bytes_to_cpy + char_len;
+      valid_bytes_to_cpy = 0;
+      continue;
+    }
+    valid_bytes_to_cpy += char_len;
+  }
+  // if there are still valid bytes to copy, do it
+  if (valid_bytes_to_cpy != 0) {

Review comment:
       If there are are no invalid chars (which should be in the majority of cases) you can skip the memcpy and return the original string.

##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1257,54 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(int64_t context, const char* text_in,
+                                                    int32_t text_len,
+                                                    const char* char_to_replace,
+                                                    int32_t char_to_replace_len,
+                                                    int32_t* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  int32_t valid_bytes_to_cpy = 0;
+  int32_t out_byte_counter = 0;
+  int32_t char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    // only memory copy the bytes when detect invalid char
+    if (char_len == 0 || text_index + char_len > text_len ||
+        !validate_utf8_following_bytes(text_in, char_len, text_index)) {
+      // define char_len = 1 to increase text_index by 1 (as ASCII char fits in 1 byte)
+      char_len = 1;
+      // first copy the valid bytes until now and then replace the invalid character
+      memcpy(ret + out_byte_counter, text_in + out_byte_counter, valid_bytes_to_cpy);
+      memcpy(ret + out_byte_counter + valid_bytes_to_cpy, char_to_replace, char_len);

Review comment:
       nit: you can do normal assignment: ret[out_byte_counter + valid_bytes_to_cpy] = char_to_replace[0]




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604801995



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,44 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 char_to_replace_len, gdv_int32* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);

Review comment:
       Since most of the time the input will be ascii, you can optimise it for that case.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604048019



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  std::string str;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += 1) {

Review comment:
       logic here is faulty. You need to increment by char_len instead of 1 after each iteration




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604801337



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,39 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0 || text_index + char_len > text_len) {

Review comment:
       it should be simple to write. You can check the logic in utf8_length function. Basically it just checks if the MSBs of the bytes is correct according to the format.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605111131



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1258,52 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 char_to_replace_len, gdv_int32* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0) {
+      memcpy(ret + out_byte_counter, char_to_replace, 1);
+      out_byte_counter += 1;
+      // define char_len = 1 to increase text_index by 1 (as ASCII char fits in 1 byte)
+      char_len = 1;
+      continue;
+    }
+    // if the char length is greater than 0, execute another validation on MSBs
+    char* invalid_char = reinterpret_cast<char*>(malloc(char_len));
+    if (text_index + char_len > text_len || validate_utf8_following_bytes(
+        text_in, char_len, text_index, invalid_char) == 0) {
+      memcpy(ret + out_byte_counter, char_to_replace, char_len);
+    } else {
+      memcpy(ret + out_byte_counter, text_in + text_index, char_len);

Review comment:
       It is currently doing a memcpy for character by character. You can keep track of valid chars across the loop and do a single memcpy at the end or after detecting invalid char. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] praveenbingo closed pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
praveenbingo closed pull request #9844:
URL: https://github.com/apache/arrow/pull/9844


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605128374



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1258,52 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,

Review comment:
       The aliases gdv_int64, gdv_int32, etc are only used to make it easy to write macros. In normal function definitions it is better to use normal types int64_t, int32_t etc. for better readability. (Can replace instances of gdv_* types in other functions with the actual types in a separate PR?)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604223982



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  std::string str;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += 1) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0 || text_index + char_len > text_len) {
+      str += char_to_replace;

Review comment:
       Corrected for not using string anymore. Now using memcpy!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605123985



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1258,52 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 char_to_replace_len, gdv_int32* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0) {
+      memcpy(ret + out_byte_counter, char_to_replace, 1);
+      out_byte_counter += 1;
+      // define char_len = 1 to increase text_index by 1 (as ASCII char fits in 1 byte)
+      char_len = 1;
+      continue;
+    }
+    // if the char length is greater than 0, execute another validation on MSBs
+    char* invalid_char = reinterpret_cast<char*>(malloc(char_len));

Review comment:
       this doesn't seem to be needed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604045553



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;

Review comment:
       How are you not using char_to_replace_len




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605181037



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1258,52 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,

Review comment:
       Sure, I will correct it for this function and then create a "story" to correct it on JIRA




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605126448



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -156,6 +156,17 @@ void set_error_for_invalid_utf(int64_t execution_context, char val) {
   free(error);
 }
 
+gdv_int32 validate_utf8_following_bytes(const char* data, gdv_int32 data_len,

Review comment:
       Can you add a FORCE_INLINE marker




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605184637



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -156,6 +156,17 @@ void set_error_for_invalid_utf(int64_t execution_context, char val) {
   free(error);
 }
 
+gdv_int32 validate_utf8_following_bytes(const char* data, gdv_int32 data_len,

Review comment:
       Corrected!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604392118



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,39 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0 || text_index + char_len > text_len) {

Review comment:
       @projjal what do you mean by "validate function"? I'm trying to find this function to apply it, but I'm not seeing it




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605180621



##########
File path: cpp/src/gandiva/precompiled/string_ops_test.cc
##########
@@ -115,6 +115,55 @@ TEST(TestStringOps, TestCharLength) {
   ctx.Reset();
 }
 
+TEST(TestStringOps, TestConvertReplaceInvalidUtf8Char) {
+  gandiva::ExecutionContext ctx;
+  uint64_t ctx_ptr = reinterpret_cast<gdv_int64>(&ctx);
+
+  // invalid utf8 (xf8 is invalid but x28 is not - x28 = '(')
+  std::string a(
+      "ok-\xf8\x28"
+      "-a");
+  auto a_in_out_len = static_cast<int>(a.length());
+  const char* a_str = convert_replace_invalid_fromUTF8_binary(
+      ctx_ptr, a.data(), a_in_out_len, "a", 1, &a_in_out_len);
+  EXPECT_EQ(std::string(a_str, a_in_out_len), "ok-a(-a");
+  EXPECT_FALSE(ctx.has_error());
+
+  // invalid utf8 (xa0 and xa1 are invalid)
+  std::string b("ok-\xa0\xa1-valid");
+  auto b_in_out_len = static_cast<int>(b.length());
+  const char* b_str = convert_replace_invalid_fromUTF8_binary(
+      ctx_ptr, b.data(), b_in_out_len, "b", 1, &b_in_out_len);
+  EXPECT_EQ(std::string(b_str, b_in_out_len), "ok-bb-valid");
+  EXPECT_FALSE(ctx.has_error());
+
+  // full valid utf8
+  std::string c("all-valid");
+  auto c_in_out_len = static_cast<int>(c.length());
+  const char* c_str = convert_replace_invalid_fromUTF8_binary(
+      ctx_ptr, c.data(), c_in_out_len, "c", 1, &c_in_out_len);
+  EXPECT_EQ(std::string(c_str, c_in_out_len), "all-valid");
+  EXPECT_FALSE(ctx.has_error());
+
+  // valid utf8 (महसुस is a 4-byte utf-8 char)

Review comment:
       Thanks, I will describe it better




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, ‘UTF8’, replacement char) function

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#issuecomment-810051836


   https://issues.apache.org/jira/browse/ARROW-12146


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605128545



##########
File path: cpp/src/gandiva/precompiled/string_ops_test.cc
##########
@@ -115,6 +115,55 @@ TEST(TestStringOps, TestCharLength) {
   ctx.Reset();
 }
 
+TEST(TestStringOps, TestConvertReplaceInvalidUtf8Char) {
+  gandiva::ExecutionContext ctx;
+  uint64_t ctx_ptr = reinterpret_cast<gdv_int64>(&ctx);
+
+  // invalid utf8 (xf8 is invalid but x28 is not - x28 = '(')
+  std::string a(
+      "ok-\xf8\x28"
+      "-a");
+  auto a_in_out_len = static_cast<int>(a.length());
+  const char* a_str = convert_replace_invalid_fromUTF8_binary(
+      ctx_ptr, a.data(), a_in_out_len, "a", 1, &a_in_out_len);
+  EXPECT_EQ(std::string(a_str, a_in_out_len), "ok-a(-a");
+  EXPECT_FALSE(ctx.has_error());
+
+  // invalid utf8 (xa0 and xa1 are invalid)
+  std::string b("ok-\xa0\xa1-valid");
+  auto b_in_out_len = static_cast<int>(b.length());
+  const char* b_str = convert_replace_invalid_fromUTF8_binary(
+      ctx_ptr, b.data(), b_in_out_len, "b", 1, &b_in_out_len);
+  EXPECT_EQ(std::string(b_str, b_in_out_len), "ok-bb-valid");
+  EXPECT_FALSE(ctx.has_error());
+
+  // full valid utf8
+  std::string c("all-valid");
+  auto c_in_out_len = static_cast<int>(c.length());
+  const char* c_str = convert_replace_invalid_fromUTF8_binary(
+      ctx_ptr, c.data(), c_in_out_len, "c", 1, &c_in_out_len);
+  EXPECT_EQ(std::string(c_str, c_in_out_len), "all-valid");
+  EXPECT_FALSE(ctx.has_error());
+
+  // valid utf8 (महसुस is a 4-byte utf-8 char)

Review comment:
       महसुस is a four character string, each character of which is likely a multibyte char




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604303465



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,39 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0 || text_index + char_len > text_len) {

Review comment:
       utf8_char_length doesn't completely validate the utf8 char. You need to call a validate function in the if statement.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604864206



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,39 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0 || text_index + char_len > text_len) {

Review comment:
       Thanks! Included validation according to utf8_length.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r605112605



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1258,52 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 char_to_replace_len, gdv_int32* out_len) {
+  if (char_to_replace_len > 1) {
+    gdv_fn_context_set_error_msg(context, "Replacement of multiple bytes not supported");
+    *out_len = 0;
+    return "";
+  }
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  gdv_int32 out_byte_counter = 0;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += char_len) {
+    char_len = utf8_char_length(text_in[text_index]);
+    if (char_len == 0) {
+      memcpy(ret + out_byte_counter, char_to_replace, 1);
+      out_byte_counter += 1;
+      // define char_len = 1 to increase text_index by 1 (as ASCII char fits in 1 byte)
+      char_len = 1;
+      continue;
+    }
+    // if the char length is greater than 0, execute another validation on MSBs
+    char* invalid_char = reinterpret_cast<char*>(malloc(char_len));

Review comment:
       please use arena malloc




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604220697



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));
+  if (ret == nullptr) {
+    gdv_fn_context_set_error_msg(context, "Could not allocate memory for output string");
+    *out_len = 0;
+    return "";
+  }
+  std::string str;
+  gdv_int32 char_len;
+  // scan the base text from left to right and increment the start pointer till
+  // looking for invalid chars to substitute
+  for (int text_index = 0; text_index < text_len; text_index += 1) {

Review comment:
       Corrected!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604049008



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;
+  char* ret = reinterpret_cast<char*>(gdv_fn_context_arena_malloc(context, *out_len));

Review comment:
       output string length can be larger than input binary length




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jpedroantunes commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
jpedroantunes commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604220433



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,33 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  *out_len = text_len;

Review comment:
       As defined by email, actually we are going to consider a single char to replace.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #9844: ARROW-12146: [C++][Gandiva] Implement CONVERT_FROM(expression, replacement char) function

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #9844:
URL: https://github.com/apache/arrow/pull/9844#discussion_r604301461



##########
File path: cpp/src/gandiva/precompiled/string_ops.cc
##########
@@ -1246,6 +1246,39 @@ const char* convert_fromUTF8_binary(gdv_int64 context, const char* bin_in, gdv_i
   return ret;
 }
 
+FORCE_INLINE
+const char* convert_replace_invalid_fromUTF8_binary(
+    gdv_int64 context, const char* text_in, gdv_int32 text_len,
+    const char* char_to_replace, gdv_int32 /*char_to_replace_len*/, gdv_int32* out_len) {
+  // actually the convert_replace function replaces the invalid bytes with a single byte
+  // so the output length will be the same as the input length

Review comment:
       If you are assuming it is single byte char, add a check to validate that the char_length of char_to_replace is 1. If its greater than one, return error saying not supported




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org