You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/10 20:18:45 UTC

[GitHub] [arrow] thisisnic opened a new pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

thisisnic opened a new pull request #10507:
URL: https://github.com/apache/arrow/pull/10507


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655844136



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+tz_aware_date <- ymd_hms("2017-01-01 00:00:12.3456789")
+tz_aware_df <- tibble::tibble(date = tz_aware_date)
+
+test_that("timezone aware timestamps are not supported",{
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})

Review comment:
       I added in the above test to demonstrate the current/expected behaviour, but I'm not totally happy with this.  I'm not sure what makes sense to do here.
   
   I need to properly catch up on the mailing list discussion on timezone aware timestamps to work out the likely possibilities and solutions, so will come back to this, but not happy with it as-is at the moment.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r657108762



##########
File path: r/R/expression.R
##########
@@ -29,8 +29,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "isoweek" = "iso_week",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "hour" = "hour",
+  "minute" = "minute"

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r657113248



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,174 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "BST")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Cannot extract components from timestamp with specific timezone"

Review comment:
       Now have added a link to the ticket

##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,174 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "BST")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Cannot extract components from timestamp with specific timezone"
+  )
+})
+
+test_that("date32 objects are not supported",{
+  
+  date <- ymd("2017-01-01")
+  df <- tibble::tibble(date = date)
+  
+  expect_error(
+    Table$create(df) %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    "Function year has no kernel matching input types"

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656451946



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,174 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "BST")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Cannot extract components from timestamp with specific timezone"
+  )
+})
+
+test_that("date32 objects are not supported",{
+  
+  date <- ymd("2017-01-01")
+  df <- tibble::tibble(date = date)
+  
+  expect_error(
+    Table$create(df) %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    "Function year has no kernel matching input types"

Review comment:
       Oh, I see below you made one. Please add a link in comment so we can find this later.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r657109569



##########
File path: r/R/expression.R
##########
@@ -29,8 +29,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656449850



##########
File path: r/R/expression.R
##########
@@ -29,8 +29,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "isoweek" = "iso_week",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "hour" = "hour",
+  "minute" = "minute"

Review comment:
       comment that second and wday are defined in dplyr-functions.R?

##########
File path: r/R/expression.R
##########
@@ -29,8 +29,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R

Review comment:
       Can you update this comment to say dplyr-functions.R?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#issuecomment-859018584


   https://issues.apache.org/jira/browse/ARROW-13022


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] rok commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
rok commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655248209



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       We could do such things in C++ as the library we use supports locale-aware [strftime](https://strftime.org/) (See [slide 26](http://schd.ws/hosted_files/cppcon2016/0f/Welcome%20To%20The%20Time%20Zone%20-%20Howard%20Hinnant%20-%20CppCon%202016.pdf). I expect we'll need it other places later too.
   
   Please make a Jira and indicate we need to discuss if this goes into R or C++.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656448695



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,74 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$second <- function(x) {
+  Expression$create("add", Expression$create("second", x), Expression$create("subsecond", x))
+}
+
+# After ARROW-13054 is completed, we can refactor this for simplicity
+# 
+# Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+# `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+# of the week is first (Sunday by default).  This Expression converts the returned
+# day of the week back to the value that would be returned by lubridate by
+# providing offset values based on the specified week_start day, and adding 1
+# so the returned value is 1-indexed instead of 0-indexed.
+# 
+# This looks complex but if you look at the lubridate implementation of wday,
+# you can see that this is the same maths, but with 1 added at the end to 
+# account for the fact that R is 1-based and the Arrow implementation is 0-based
+# 
+# overall formula to convert from arrow::wday to lubridate::wday is:
+#  (3 +wday(day) + (5 - start)) %% 7) + 1
+# 
+# To calculate e1 %% e2, we can rearrange it as:
+# {e1 - e2 * ( e1 %/% e2 )}
+# 
+# And e1 %/% e2 can itself be done via casting e1/e2 to an integer
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  
+  # e1 = 3 +wday(day) + (5 - start)
+  e1 = Expression$create(
+    "add_checked",
+    # 3 + wday(date)
+    Expression$create("add_checked",
+                      Expression$scalar(3),
+                      Expression$create("day_of_week", x)
+    ),
+    # (5 - start)
+    Expression$create("subtract_checked",
+                      Expression$scalar(5),
+                      Expression$scalar(week_start)
+    )
+  )
+  
+  e2 = Expression$scalar(7)
+  
+  # (e1 - e2 * ( e1 %/% e2 )) + 1
+  Expression$create(
+    "add_checked",
+    Expression$scalar(1),                
+    # e1 - e2 * ( e1 %/% e2 )
+    Expression$create(
+      "subtract_checked",
+      # e1
+      e1,
+      # e2 * ( e1 %/% e2 )
+      Expression$create("multiply_checked",
+        e2,
+        # e1 %/% e2; because as.integer(x/y) == x%/%y
+        Expression$create(
+          "cast",
+          Expression$create("divide_checked", e1, e2),
+          options = cast_options(to_type = int32(), allow_float_truncate = TRUE,

Review comment:
       If you make all of the scalars integers (e.g. 1L), do you still need to cast?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655256202



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       Thanks, will do!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655848010



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+tz_aware_date <- ymd_hms("2017-01-01 00:00:12.3456789")
+tz_aware_df <- tibble::tibble(date = tz_aware_date)
+
+test_that("timezone aware timestamps are not supported",{
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})

Review comment:
       Not entirely sure, but I think when this is merged, it will allow the above issue to be resolved: https://issues.apache.org/jira/browse/ARROW-12980




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656553130



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,174 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "BST")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Cannot extract components from timestamp with specific timezone"

Review comment:
       Got it, thanks, I wasn't sure where the check was happening (and great that there's a PR that would make this restriction go away anyway)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655959918



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+tz_aware_date <- ymd_hms("2017-01-01 00:00:12.3456789")
+tz_aware_df <- tibble::tibble(date = tz_aware_date)
+
+test_that("timezone aware timestamps are not supported",{
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})

Review comment:
       I think this is fine for now, since we this error on purpose in the initial PR to limit scope. And this functionality will indeed be added in a next PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r650076060



##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       IDK that we should truncate the data like this. A slight (technical) API difference is probably better than throwing away precision.

##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- ymd_hms("1987-10-09 23:00:00", tz = NULL)
+test_df <- tibble::tibble(date = test_date)
+
+test_that("extract datetime components from date", {
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = isoyear(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = quarter(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = month(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = wday(date, week_start = 3)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_warning(
+    test_df %>%
+      Table$create() %>%
+      mutate(x = wday(date, label = TRUE)) %>%
+      collect(),
+    regexp = "Label argument not supported by Arrow; pulling data into R"
+  )
+  
+  expect_warning(
+    test_df %>%
+      Table$create() %>%
+      mutate(x = wday(date, locale = Sys.getlocale("LC_TIME"))) %>%
+      collect(),
+    regexp = 'Expression wday(date, locale = Sys.getlocale("LC_TIME")) not supported in Arrow; pulling data into R',
+    fixed = TRUE
+  )

Review comment:
       Yeah I think we can do this more unit-testy version where appropriate; we test elsewhere that the errors we throw in the nse_funcs get handled by filter/mutate correctly.

##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       What is the label argument? Is it something we should support? I think it's good to document these unsupported exceptions in comments, to note either why Arrow should behave differently, or what Jira issue exists to remove the exception.

##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  offset <- get_date_offset(week_start)
+  Expression$create("add", Expression$create("day_of_week", x), Expression$scalar(offset))
+}
+
+#' Get date offset
+#' 
+#' Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+#' `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+#' of the week is first (Sunday by default).  This function converts the returned

Review comment:
       Please leave a comment with that JIRA issue somewhere in this code that we expect to remove.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ianmcook commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
ianmcook commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r650057966



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- ymd_hms("1987-10-09 23:00:00", tz = NULL)
+test_df <- tibble::tibble(date = test_date)
+
+test_that("extract datetime components from date", {
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = isoyear(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = quarter(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = month(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = wday(date, week_start = 3)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_warning(
+    test_df %>%
+      Table$create() %>%
+      mutate(x = wday(date, label = TRUE)) %>%
+      collect(),
+    regexp = "Label argument not supported by Arrow; pulling data into R"
+  )
+  
+  expect_warning(
+    test_df %>%
+      Table$create() %>%
+      mutate(x = wday(date, locale = Sys.getlocale("LC_TIME"))) %>%
+      collect(),
+    regexp = 'Expression wday(date, locale = Sys.getlocale("LC_TIME")) not supported in Arrow; pulling data into R',
+    fixed = TRUE
+  )

Review comment:
       For cases in which an error is thrown and then converted to a warning, I think we prefer to write the tests like this:
   ```r
   x <- Expression$field_ref("x")
   expect_error(
     nse_funcs$wday(x, label = TRUE),
     "Label argument not supported by Arrow"
   )
   ```
   I'm not sure how important this is. @nealrichardson does it matter to you?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656451343



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,174 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "BST")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Cannot extract components from timestamp with specific timezone"

Review comment:
       (1) wouldn't they work if timezone = UTC?
   (2) can you link to the JIRA here in a comment?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655848143



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,175 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- ymd_hms("2017-01-01")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})
+
+test_that("date32 objects are not supported",{
+  
+  date <- ymd("2017-01-01")
+  df <- tibble::tibble(date = date)
+  
+  expect_error(
+    Table$create(df) %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    "Function year has no kernel matching input types"
+  )
+})
+

Review comment:
       https://issues.apache.org/jira/browse/ARROW-13138




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655259199



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       https://issues.apache.org/jira/browse/ARROW-13133




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson closed pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson closed pull request #10507:
URL: https://github.com/apache/arrow/pull/10507


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655221384



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       The label argument allows the return of the day of the week in the correct language.
   
   In `lubridate`, there's a whole lot of work around defining the locale and then setting the weekday names appropriately, though essentially, it just uses the format function to extract the relevant date.
   	
   I think that we wouldn't want to implement this at the C++ level as it doesn't feel like it fits there as it's doing quite a lot more work than calculating a value, (i.e. returning locale-specific dates), and we could still achieve this with good performance at the R level by implementing this functionality using the ifelse kernel once its R bindings are implemented.
   
   How about if I leave it as unsupported now, but create a ticket to implement it later, that lists the ifelse R binding as a dependency?

##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       Thanks, will do!

##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       https://issues.apache.org/jira/browse/ARROW-13133

##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+tz_aware_date <- ymd_hms("2017-01-01 00:00:12.3456789")
+tz_aware_df <- tibble::tibble(date = tz_aware_date)
+
+test_that("timezone aware timestamps are not supported",{
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})

Review comment:
       I added in the above test to demonstrate the current/expected behaviour, but I'm not totally happy with this.  I'm not sure what makes sense to do here.
   
   I need to properly catch up on the mailing list discussion on timezone aware timestamps to work out the likely possibilities and solutions, so will come back to this, but not happy with it as-is at the moment.

##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,175 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- ymd_hms("2017-01-01")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})
+
+test_that("date32 objects are not supported",{
+  
+  date <- ymd("2017-01-01")
+  df <- tibble::tibble(date = date)
+  
+  expect_error(
+    Table$create(df) %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    "Function year has no kernel matching input types"
+  )
+})
+

Review comment:
       Will open a JIRA to implement for these object types.

##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+tz_aware_date <- ymd_hms("2017-01-01 00:00:12.3456789")
+tz_aware_df <- tibble::tibble(date = tz_aware_date)
+
+test_that("timezone aware timestamps are not supported",{
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})

Review comment:
       Not entirely sure, but I think when this is merged, it will allow the above issue to be resolved: https://issues.apache.org/jira/browse/ARROW-12980

##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,175 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- ymd_hms("2017-01-01")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})
+
+test_that("date32 objects are not supported",{
+  
+  date <- ymd("2017-01-01")
+  df <- tibble::tibble(date = date)
+  
+  expect_error(
+    Table$create(df) %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    "Function year has no kernel matching input types"
+  )
+})
+

Review comment:
       https://issues.apache.org/jira/browse/ARROW-13138




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656528973



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,174 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "BST")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Cannot extract components from timestamp with specific timezone"

Review comment:
       > (1) wouldn't they work if timezone = UTC?
   
   We could have made it work already for UTC, but at the moment it simply checks if "a" timezone is set in the C++ code (there is an open PR to add timezone support)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655221384



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       The label argument allows the return of the day of the week in the correct language.
   
   In `lubridate`, there's a whole lot of work around defining the locale and then setting the weekday names appropriately, though essentially, it just uses the format function to extract the relevant date.
   	
   I think that we wouldn't want to implement this at the C++ level as it doesn't feel like it fits there as it's doing quite a lot more work than calculating a value, (i.e. returning locale-specific dates), and we could still achieve this with good performance at the R level by implementing this functionality using the ifelse kernel once its R bindings are implemented.
   
   How about if I leave it as unsupported now, but create a ticket to implement it later, that lists the ifelse R binding as a dependency?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r657103705



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       Yup, on it!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656447630



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       Can you leave a comment explaining this and linking to the JIRA issue?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] rok commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
rok commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655248209



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       We could do such things in C++ as the library we use supports locale-aware [strftime](https://strftime.org/) (See [slide 26](http://schd.ws/hosted_files/cppcon2016/0f/Welcome%20To%20The%20Time%20Zone%20-%20Howard%20Hinnant%20-%20CppCon%202016.pdf). I expect we'll need it other places later too.
   
   Please make a Jira and indicate we need to discuss if this goes into R or C++.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656449141



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,74 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$second <- function(x) {
+  Expression$create("add", Expression$create("second", x), Expression$create("subsecond", x))
+}
+
+# After ARROW-13054 is completed, we can refactor this for simplicity
+# 
+# Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+# `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+# of the week is first (Sunday by default).  This Expression converts the returned
+# day of the week back to the value that would be returned by lubridate by
+# providing offset values based on the specified week_start day, and adding 1
+# so the returned value is 1-indexed instead of 0-indexed.
+# 
+# This looks complex but if you look at the lubridate implementation of wday,
+# you can see that this is the same maths, but with 1 added at the end to 
+# account for the fact that R is 1-based and the Arrow implementation is 0-based
+# 
+# overall formula to convert from arrow::wday to lubridate::wday is:
+#  (3 +wday(day) + (5 - start)) %% 7) + 1
+# 
+# To calculate e1 %% e2, we can rearrange it as:
+# {e1 - e2 * ( e1 %/% e2 )}
+# 
+# And e1 %/% e2 can itself be done via casting e1/e2 to an integer
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  
+  # e1 = 3 +wday(day) + (5 - start)
+  e1 = Expression$create(
+    "add_checked",
+    # 3 + wday(date)
+    Expression$create("add_checked",
+                      Expression$scalar(3),
+                      Expression$create("day_of_week", x)
+    ),
+    # (5 - start)
+    Expression$create("subtract_checked",
+                      Expression$scalar(5),
+                      Expression$scalar(week_start)
+    )
+  )
+  
+  e2 = Expression$scalar(7)
+  
+  # (e1 - e2 * ( e1 %/% e2 )) + 1

Review comment:
       Did you try doing exactly this expression? I would think it should just work because `Ops.Expression` is defined for all of those.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r657104439



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,74 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$second <- function(x) {
+  Expression$create("add", Expression$create("second", x), Expression$create("subsecond", x))
+}
+
+# After ARROW-13054 is completed, we can refactor this for simplicity
+# 
+# Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+# `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+# of the week is first (Sunday by default).  This Expression converts the returned
+# day of the week back to the value that would be returned by lubridate by
+# providing offset values based on the specified week_start day, and adding 1
+# so the returned value is 1-indexed instead of 0-indexed.
+# 
+# This looks complex but if you look at the lubridate implementation of wday,
+# you can see that this is the same maths, but with 1 added at the end to 
+# account for the fact that R is 1-based and the Arrow implementation is 0-based
+# 
+# overall formula to convert from arrow::wday to lubridate::wday is:
+#  (3 +wday(day) + (5 - start)) %% 7) + 1
+# 
+# To calculate e1 %% e2, we can rearrange it as:
+# {e1 - e2 * ( e1 %/% e2 )}
+# 
+# And e1 %/% e2 can itself be done via casting e1/e2 to an integer
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  
+  # e1 = 3 +wday(day) + (5 - start)
+  e1 = Expression$create(
+    "add_checked",
+    # 3 + wday(date)
+    Expression$create("add_checked",
+                      Expression$scalar(3),
+                      Expression$create("day_of_week", x)
+    ),
+    # (5 - start)
+    Expression$create("subtract_checked",
+                      Expression$scalar(5),
+                      Expression$scalar(week_start)
+    )
+  )
+  
+  e2 = Expression$scalar(7)
+  
+  # (e1 - e2 * ( e1 %/% e2 )) + 1

Review comment:
       Awesome, I had no idea, deleted a lot of unnecessary lines of code now, thanks @nealrichardson ! 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r650738642



##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       Agreed, I would also say that it's not because lubridate does not support nanoseconds that *if* you actually have nanoseconds (which is possible since arrow supports it) those should be discarded. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r657236041



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,30 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$second <- function(x) {
+  Expression$create("add", Expression$create("second", x), Expression$create("subsecond", x))
+}
+
+# After ARROW-13054 is completed, we can refactor this for simplicity
+# 
+# Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+# `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+# of the week is first (Sunday by default).  This Expression converts the returned
+# day of the week back to the value that would be returned by lubridate by
+# providing offset values based on the specified week_start day, and adding 1
+# so the returned value is 1-indexed instead of 0-indexed.
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  
+  # The "day_of_week" compute function returns numeric days of week and not locale-aware strftime
+  # When the ticket below is resolved, we should be able to support the label argument
+  # https://issues.apache.org/jira/browse/ARROW-13133
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  
+  # overall formula to convert from arrow::wday to lubridate::wday is:
+  #  ((wday(day) - start + 8) %% 7) + 1
+  ((Expression$create("day_of_week", x)  -  Expression$scalar(week_start) + 8) %%  7) + 1

Review comment:
       ```suggestion
     ((Expression$create("day_of_week", x) - Expression$scalar(week_start) + 8) %% 7) + 1
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r657098046



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  offset <- get_date_offset(week_start)
+  Expression$create("add", Expression$create("day_of_week", x), Expression$scalar(offset))
+}
+
+#' Get date offset
+#' 
+#' Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+#' `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+#' of the week is first (Sunday by default).  This function converts the returned

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] rok commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
rok commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r649895508



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  offset <- get_date_offset(week_start)
+  Expression$create("add", Expression$create("day_of_week", x), Expression$scalar(offset))
+}
+
+#' Get date offset
+#' 
+#' Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+#' `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+#' of the week is first (Sunday by default).  This function converts the returned

Review comment:
       We could add options to the C++ kernel to enable different behaviors there.

##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  offset <- get_date_offset(week_start)
+  Expression$create("add", Expression$create("day_of_week", x), Expression$scalar(offset))
+}
+
+#' Get date offset
+#' 
+#' Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+#' `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+#' of the week is first (Sunday by default).  This function converts the returned

Review comment:
       If we go that way it would probably be best to have another Jira for "TemporalOptions". It's probably best you proceed with the workaround and we loop back to this later.

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       What about nanoseconds?
   ```R
   > second(ymd_hms("2011-06-04 12:00:01.123456789"))
   [1] 1.123457
   ```
   
   Arrow would probably return `1.123456789`.

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       I think so too. So probably it should be "`second = second + round(subsecond, 6)`" to match that behaviour?

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       I think so too. So probably it should be "`second = second + round(subsecond, 6)`" to match that behavior?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r649723619



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  offset <- get_date_offset(week_start)
+  Expression$create("add", Expression$create("day_of_week", x), Expression$scalar(offset))
+}
+
+#' Get date offset
+#' 
+#' Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+#' `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+#' of the week is first (Sunday by default).  This function converts the returned

Review comment:
       I was going to mention that we actually also have an "ISO weekday" in the C++ kernels, but only as field in the `iso_calendar` kernel, and not as separate one. We could also add `iso_day_of_week` if that helps, or even just make `day_of_week` follow the ISO conventions. Because the 0 (monday) - 6 (sunday) might be something Python specific.
   
   But then looked at what C++ does, and there the default is actually 0 (sunday) - 6 (saturday), so yet something else ;) (although I see that also Postgres uses that for `dow`)
   
   In the end, once you have it in one form, it's an easy conversion to any of the other of course, so it might not matter that much.
   
   

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",

Review comment:
       "hour" is missing here?

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       Note that second might do something different. I think "second" in lubridate is the equivalent of "second + subsecond" in arrow

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       What do you mean exactly with rounding? A quick try gives me:
   
   ```
   > second(ymd_hms("2011-06-04 12:00:01.123456"))
   [1] 1.123456
   ```
   
   which seems to give all decimals

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       I don't think lubridate / R supports nanosecond resolution




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656451510



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,174 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "BST")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Cannot extract components from timestamp with specific timezone"
+  )
+})
+
+test_that("date32 objects are not supported",{
+  
+  date <- ymd("2017-01-01")
+  df <- tibble::tibble(date = date)
+  
+  expect_error(
+    Table$create(df) %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    "Function year has no kernel matching input types"

Review comment:
       Is there a JIRA for this? This should be supported.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r655844664



##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,175 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- as.POSIXct("2017-01-01 00:00:12.3456789", tz = "")
+test_df <- tibble::tibble(date = test_date)
+
+test_that("timezone aware timestamps are not supported",{
+  
+  tz_aware_date <- ymd_hms("2017-01-01")
+  tz_aware_df <- tibble::tibble(date = tz_aware_date)
+  
+  x <- Expression$field_ref("x")
+  expect_error(
+    Table$create(tz_aware_df) %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    "Timezone aware timestamps not supported"
+  )
+})
+
+test_that("date32 objects are not supported",{
+  
+  date <- ymd("2017-01-01")
+  df <- tibble::tibble(date = date)
+  
+  expect_error(
+    Table$create(df) %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    "Function year has no kernel matching input types"
+  )
+})
+

Review comment:
       Will open a JIRA to implement for these object types.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, hour, minute, and second functions

Posted by GitBox <gi...@apache.org>.
nealrichardson commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r656449431



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,74 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$second <- function(x) {
+  Expression$create("add", Expression$create("second", x), Expression$create("subsecond", x))
+}
+
+# After ARROW-13054 is completed, we can refactor this for simplicity
+# 
+# Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+# `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+# of the week is first (Sunday by default).  This Expression converts the returned
+# day of the week back to the value that would be returned by lubridate by
+# providing offset values based on the specified week_start day, and adding 1
+# so the returned value is 1-indexed instead of 0-indexed.
+# 
+# This looks complex but if you look at the lubridate implementation of wday,
+# you can see that this is the same maths, but with 1 added at the end to 
+# account for the fact that R is 1-based and the Arrow implementation is 0-based
+# 
+# overall formula to convert from arrow::wday to lubridate::wday is:
+#  (3 +wday(day) + (5 - start)) %% 7) + 1
+# 
+# To calculate e1 %% e2, we can rearrange it as:
+# {e1 - e2 * ( e1 %/% e2 )}
+# 
+# And e1 %/% e2 can itself be done via casting e1/e2 to an integer
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  
+  # e1 = 3 +wday(day) + (5 - start)
+  e1 = Expression$create(

Review comment:
       `<-`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] thisisnic commented on a change in pull request #10507: ARROW-13022: [R] bindings for lubridate's year, isoyear, quarter, month, day, wday, yday, isoweek, minute, and second functions

Posted by GitBox <gi...@apache.org>.
thisisnic commented on a change in pull request #10507:
URL: https://github.com/apache/arrow/pull/10507#discussion_r649907028



##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  offset <- get_date_offset(week_start)
+  Expression$create("add", Expression$create("day_of_week", x), Expression$scalar(offset))
+}
+
+#' Get date offset
+#' 
+#' Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+#' `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+#' of the week is first (Sunday by default).  This function converts the returned

Review comment:
       That probably makes more sense than my workaround here actually.

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       Good catch.  Hmm, I think this might be more complicated than that actually, as lubridate rounds it to 1 decimal place, using R's odd-even rounding whereas Arrow doesn't do the rounding, so I'm not sure if this can be done without rounding being implemented (see https://issues.apache.org/jira/browse/ARROW-12744)

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",

Review comment:
       Oops, will add in now.

##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  offset <- get_date_offset(week_start)
+  Expression$create("add", Expression$create("day_of_week", x), Expression$scalar(offset))
+}
+
+#' Get date offset
+#' 
+#' Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+#' `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+#' of the week is first (Sunday by default).  This function converts the returned

Review comment:
       OK, will do, and will open a JIRA, thanks!

##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")
+  }
+  offset <- get_date_offset(week_start)
+  Expression$create("add", Expression$create("day_of_week", x), Expression$scalar(offset))
+}
+
+#' Get date offset
+#' 
+#' Arrow's `day_of_week` kernel counts from 0 (Monday) to 6 (Sunday), whereas
+#' `lubridate::wday` counts from 1 to 7, and allows users to specify which day
+#' of the week is first (Sunday by default).  This function converts the returned

Review comment:
       https://issues.apache.org/jira/browse/ARROW-13054

##########
File path: r/R/expression.R
##########
@@ -28,8 +28,17 @@
   # stringr spellings of those
   "str_length" = "utf8_length",
   "str_to_lower" = "utf8_lower",
-  "str_to_upper" = "utf8_upper"
+  "str_to_upper" = "utf8_upper",
   # str_trim is defined in dplyr.R
+  "year" = "year",
+  "isoyear" = "iso_year",
+  "quarter" = "quarter",
+  "month" = "month",
+  "day" = "day",
+  "yday" = "day_of_year",
+  "isoweek" = "iso_week",
+  "minute" = "minute",
+  "second" = "second"

Review comment:
       So it does; I think I must have been getting mixed up with something I'd changed when I needed to update a test I wrote, never mind,

##########
File path: r/tests/testthat/test-dplyr-lubridate.R
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+library(lubridate)
+library(dplyr)
+
+test_date <- ymd_hms("1987-10-09 23:00:00", tz = NULL)
+test_df <- tibble::tibble(date = test_date)
+
+test_that("extract datetime components from date", {
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = year(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = isoyear(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = quarter(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = month(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = wday(date)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_dplyr_equal(
+    input %>%
+      mutate(x = wday(date, week_start = 3)) %>%
+      collect(),
+    test_df
+  )
+  
+  expect_warning(
+    test_df %>%
+      Table$create() %>%
+      mutate(x = wday(date, label = TRUE)) %>%
+      collect(),
+    regexp = "Label argument not supported by Arrow; pulling data into R"
+  )
+  
+  expect_warning(
+    test_df %>%
+      Table$create() %>%
+      mutate(x = wday(date, locale = Sys.getlocale("LC_TIME"))) %>%
+      collect(),
+    regexp = 'Expression wday(date, locale = Sys.getlocale("LC_TIME")) not supported in Arrow; pulling data into R',
+    fixed = TRUE
+  )

Review comment:
       Good point, I think I was just copying and pasting from the older way of doing things.  Whether it matters or not to @nealrichardson, I think that way is a lot clearer so I'll update them!

##########
File path: r/R/dplyr-functions.R
##########
@@ -442,3 +442,37 @@ nse_funcs$strptime <- function(x, format = "%Y-%m-%d %H:%M:%S", tz = NULL, unit
 
   Expression$create("strptime", x, options = list(format = format, unit = unit))
 }
+
+nse_funcs$wday <- function(x, label = FALSE, abbr = TRUE, week_start = getOption("lubridate.week.start", 7)) {
+  if (label) {
+    arrow_not_supported("Label argument")

Review comment:
       The label argument is whether the day is displayed as a character string or a number.  Maybe it is something we should support; shall I open a ticket?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org