You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/07/02 15:25:12 UTC
[GitHub] [arrow] pitrou commented on a change in pull request #7611: ARROW-3308: [R] Convert R character vector with data exceeding 2GB to Large type
pitrou commented on a change in pull request #7611:
URL: https://github.com/apache/arrow/pull/7611#discussion_r449081453
##########
File path: r/src/array_from_vector.cpp
##########
@@ -1150,6 +1150,25 @@ std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<REALSXP>(SEXP x) {
return float64();
}
+template <>
+std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<STRSXP>(SEXP x) {
+ // See how big the character vector is
+ R_xlen_t n = XLENGTH(x);
+ int64_t size = 0;
+ for (R_xlen_t i = 0; i < n; i++) {
+ SEXP string_i = STRING_ELT(x, i);
+ if (string_i != NA_STRING) {
+ size += XLENGTH(Rf_mkCharCE(Rf_translateCharUTF8(string_i), CE_UTF8));
+ }
+ if (size > 2147483646) {
+ // Exceeds 2GB capacity of utf8 type, so use large
+ return large_utf8();
+ }
+ }
+
+ return utf8();
+}
Review comment:
Yes, ideally we'd have a `AdaptiveStringBuilder`-like clas.
##########
File path: r/src/array_from_vector.cpp
##########
@@ -1150,6 +1150,25 @@ std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<REALSXP>(SEXP x) {
return float64();
}
+template <>
+std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<STRSXP>(SEXP x) {
+ // See how big the character vector is
+ R_xlen_t n = XLENGTH(x);
+ int64_t size = 0;
+ for (R_xlen_t i = 0; i < n; i++) {
+ SEXP string_i = STRING_ELT(x, i);
+ if (string_i != NA_STRING) {
+ size += XLENGTH(Rf_mkCharCE(Rf_translateCharUTF8(string_i), CE_UTF8));
+ }
+ if (size > 2147483646) {
+ // Exceeds 2GB capacity of utf8 type, so use large
+ return large_utf8();
+ }
+ }
+
+ return utf8();
+}
Review comment:
Yes, ideally we'd have a `AdaptiveStringBuilder`-like class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org