[GitHub] [arrow] nealrichardson commented on a change in pull request #7611: ARROW-3308: [R] Convert R character vector with data exceeding 2GB to Large type

[GitBox <gi...@apache.org>]

nealrichardson commented on a change in pull request #7611:
URL: https://github.com/apache/arrow/pull/7611#discussion_r449083406



##########
File path: r/src/array_from_vector.cpp
##########
@@ -1150,6 +1150,25 @@ std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<REALSXP>(SEXP x) {
   return float64();
 }
 
+template <>
+std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<STRSXP>(SEXP x) {
+  // See how big the character vector is
+  R_xlen_t n = XLENGTH(x);
+  int64_t size = 0;
+  for (R_xlen_t i = 0; i < n; i++) {
+    SEXP string_i = STRING_ELT(x, i);
+    if (string_i != NA_STRING) {
+      size += XLENGTH(Rf_mkCharCE(Rf_translateCharUTF8(string_i), CE_UTF8));
+    }
+    if (size > 2147483646) {
+      // Exceeds 2GB capacity of utf8 type, so use large
+      return large_utf8();
+    }
+  }
+
+  return utf8();
+}

Review comment:
       Here's the definition of `translateCharUTF8`: https://github.com/wch/r-source/blob/122dcf452ec5eacdd66e165457985bded1af4fee/src/main/sysutils.c#L1085
   
   If the string is already UTF-8 or ASCII, it does nothing. So FWIW the performance hit would be exceedingly rare except on Windows. That said, it would be better if we could ingest a character vector without iterating through it and calling `iconv` 3 times.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org