You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/06/21 17:09:32 UTC

[GitHub] [arrow] projjal opened a new pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

projjal opened a new pull request #7504:
URL: https://github.com/apache/arrow/pull/7504


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#discussion_r443432148



##########
File path: cpp/src/gandiva/to_date_holder.cc
##########
@@ -83,7 +83,7 @@ int64_t ToDateHolder::operator()(ExecutionContext* context, const std::string& d
   // 1. processes date that do not match the format.
   // 2. does not process time in format +08:00 (or) id.
   int64_t seconds_since_epoch = 0;
-  if (!::arrow::internal::ParseTimestampStrptime(
+  if (!::arrow::internal::ParseTimestampStrptimeFromCString(
           data.c_str(), data.length(), pattern_.c_str(),

Review comment:
       @pitrou here data.c_str() returns null-terminated string which we pass to parsetimestamp method




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on pull request #7504: ARROW-9193: [C++] Avoid spurious intermediate string copy in ToDateHolder

Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#issuecomment-647544705


   @projjal Can you fix the Windows build failures (see AppVeyor build):
   ```
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(55): error C2220: warning treated as error - no 'object' file generated
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(55): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(60): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(65): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(79): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(84): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(96): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(101): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(106): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(111): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(117): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   C:/projects/arrow/cpp/src/gandiva/to_date_holder_test.cc(131): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
   ```
   ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on a change in pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#discussion_r443432735



##########
File path: cpp/src/gandiva/to_date_holder.cc
##########
@@ -83,7 +83,7 @@ int64_t ToDateHolder::operator()(ExecutionContext* context, const std::string& d
   // 1. processes date that do not match the format.
   // 2. does not process time in format +08:00 (or) id.
   int64_t seconds_since_epoch = 0;
-  if (!::arrow::internal::ParseTimestampStrptime(
+  if (!::arrow::internal::ParseTimestampStrptimeFromCString(
           data.c_str(), data.length(), pattern_.c_str(),

Review comment:
       Understood. But where does that data come from? Why do you have a `std::string`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#discussion_r443442994



##########
File path: cpp/src/gandiva/to_date_holder.cc
##########
@@ -83,7 +83,7 @@ int64_t ToDateHolder::operator()(ExecutionContext* context, const std::string& d
   // 1. processes date that do not match the format.
   // 2. does not process time in format +08:00 (or) id.
   int64_t seconds_since_epoch = 0;
-  if (!::arrow::internal::ParseTimestampStrptime(
+  if (!::arrow::internal::ParseTimestampStrptimeFromCString(
           data.c_str(), data.length(), pattern_.c_str(),

Review comment:
       Makes sense. I will update the patch to pass the data buffer directly to the orginal parsetimestamp method. The new method won't be necessary then i guess.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou closed pull request #7504: ARROW-9193: [C++] Avoid spurious intermediate string copy in ToDateHolder

Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #7504:
URL: https://github.com/apache/arrow/pull/7504


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] fsaintjacques commented on pull request #7504: ARROW-9193: [C++] Avoid spurious intermediate string copy in ToDateHolder

Posted by GitBox <gi...@apache.org>.
fsaintjacques commented on pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#issuecomment-647596137


   Any reason not to use a `util::string_view`?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#issuecomment-647397849


   @projjal Where do the null-terminated strings come from? Doesn't Gandiva operate on Arrow data?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
projjal commented on pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#issuecomment-647155468


   Hi @wesm can you take a look at this. It seems that the ParseTimestampStrptime function was recently changed from taking c-style null terminated string input to c++ style. If the input string was already null terminated then this method significantly slows down due to extra allocation and copies from constructor and c_str(). From the benchmark test that I have added it seems that it gets about 2-2.5x slower. Would it make sense to add another method that explicitly takes null-terminated string as input to avoid these copies.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on a change in pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
projjal commented on a change in pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#discussion_r443434522



##########
File path: cpp/src/gandiva/to_date_holder.cc
##########
@@ -83,7 +83,7 @@ int64_t ToDateHolder::operator()(ExecutionContext* context, const std::string& d
   // 1. processes date that do not match the format.
   // 2. does not process time in format +08:00 (or) id.
   int64_t seconds_since_epoch = 0;
-  if (!::arrow::internal::ParseTimestampStrptime(
+  if (!::arrow::internal::ParseTimestampStrptimeFromCString(
           data.c_str(), data.length(), pattern_.c_str(),

Review comment:
       It is created here : https://github.com/apache/arrow/blob/295e2c11642f1b32aff1473de3f89ee02baada8a/cpp/src/gandiva/gdv_function_stubs.cc#L60




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on a change in pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#discussion_r443428904



##########
File path: cpp/src/arrow/util/value_parsing.h
##########
@@ -565,6 +565,39 @@ static inline bool ParseTimestampStrptime(const char* buf, size_t length,
   return true;
 }
 
+/// \brief Returns time since the UNIX epoch in the requested unit. Takes null terminated
+/// buffer as argument
+static inline bool ParseTimestampStrptimeFromCString(const char* buf, size_t length,

Review comment:
       Can you please refactor these two functions to avoid duplicated code?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] wesm commented on pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#issuecomment-647162905


   Seems reasonable, I or @pitrou will have to take a closer look. Can we centralize the benchmarks, though, so we don't have benchmarks for stuff in src/arrow/util in src/gandiva?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] projjal commented on pull request #7504: ARROW-9193: [C++] Avoid spurious intermediate string copy in ToDateHolder

Posted by GitBox <gi...@apache.org>.
projjal commented on pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#issuecomment-647691097


   > Any reason not to use a `util::string_view`?
   
   Since the toDate function here takes a char array and just passes it onto the parseTimestamp method, using a util::string_view is probably not required.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#issuecomment-647156126


   https://issues.apache.org/jira/browse/ARROW-9193


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on a change in pull request #7504: ARROW-9193: [C++] Add method to parse date from null-terminated string

Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7504:
URL: https://github.com/apache/arrow/pull/7504#discussion_r443441540



##########
File path: cpp/src/gandiva/to_date_holder.cc
##########
@@ -83,7 +83,7 @@ int64_t ToDateHolder::operator()(ExecutionContext* context, const std::string& d
   // 1. processes date that do not match the format.
   // 2. does not process time in format +08:00 (or) id.
   int64_t seconds_since_epoch = 0;
-  if (!::arrow::internal::ParseTimestampStrptime(
+  if (!::arrow::internal::ParseTimestampStrptimeFromCString(
           data.c_str(), data.length(), pattern_.c_str(),

Review comment:
       Ah. Well, if you're concerned about performance, you shouldn't create a `std::string` there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org