You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/06 22:07:53 UTC

[GitHub] [spark] MrPowers opened a new pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

MrPowers opened a new pull request #31073:
URL: https://github.com/apache/spark/pull/31073


   ### What changes were proposed in this pull request?
   
   This pull request exposes the `make_interval` function, [as suggested here](https://github.com/apache/spark/pull/31000#pullrequestreview-560812433), and as agreed to [here](https://github.com/apache/spark/pull/31000#issuecomment-754856820) and [here](https://github.com/apache/spark/pull/31000#issuecomment-755040234).
   
   This powerful little function allows for idiomatic datetime arithmetic via the Scala API:
   
   ```scala
   // add two hours
   df.withColumn("plus_2_hours", col("first_datetime") + make_interval(hours = lit(2)))
   
   // subtract one week and 30 seconds
   col("d") - make_interval(weeks = lit(1), secs = lit(30))
   ```
   
   The `make_interval` [SQL function](https://github.com/apache/spark/pull/26446) already exists.
   
   Here is [this JIRA ticket](https://issues.apache.org/jira/browse/SPARK-33995) for this PR.
   
   ### Why are the changes needed?
   
   The Spark API makes it easy to perform datetime addition / subtraction with months (`add_months`) and days (`date_add`).  Users need to write code like this to perform datetime addition with years, weeks, hours, minutes, or seconds:
   
   ```scala
   df.withColumn("plus_2_hours", expr("first_datetime + INTERVAL 2 hours"))
   ```
   
   We don't want to force users to manipulate SQL strings when they're using the Scala API.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, this PR adds `make_interval` to the `org.apache.spark.sql.functions` API.  
   
   This single function will benefit a lot of users.  It's a small increase in the surface of the API for a big gain.
   
   ### How was this patch tested?
   
   This was tested via unit tests.
   
   cc: @MaxGekk 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553431862



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       @HyukjinKwon - thank you for the code review.  The default arguments allow for this nice syntax:
   
   ```scala
   df.withColumn("plus_2_hours", col("first_datetime") + make_interval(hours = lit(2)))
   ```
   
   Without the default arguments, we'll need to invoke the method with a lot of arguments:
   
   ```scala
   val twoHourInterval = make_interval(lit(0), lit(0), lit(0), lit(0), lit(2), lit(0), lit(0))
   df.withColumn("plus_2_hours", col("first_datetime") + twoHourInterval)
   ```
   
   We could overload the method and allow for both syntaxes (using a little trick with the last argument that's optional).
   
   ```scala
   def make_interval(
       years: Column,
       months: Column,
       weeks: Column,
       days: Column,
       hours: Column,
       mins: Column,
       secs: Column,
       failOnError: Boolean): Column = withExpr {
     MakeInterval(
       years.expr, months.expr, weeks.expr, days.expr, hours.expr, mins.expr, secs.expr, failOnError)
   }
   ```
   
   This would allow Scala users to either manually list 8 arguments (e.g. `make_interval(lit(0), lit(0), lit(0), lit(0), lit(2), lit(0), lit(0), true)` or use the clean default syntax (e.g. `make_interval(hours = lit(2))`).  Would this give the Java users a usable interface?
   
   I'm also fine removing the default values if that's what's necessary to make this method compatible with Java.  @MaxGekk - open to your thoughts too cause I know you were originally in favor [of the arguments with default values approach](https://issues.apache.org/jira/browse/SPARK-33995?focusedCommentId=17258718&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17258718).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755986593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133776/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jaceklaskowski commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
jaceklaskowski commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553969891



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(

Review comment:
       I recommend creating a JIRA issue for the other langs and keep focusing on one language only. That will make code review easier (for one-language-only eyes like mine).

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       > Can we avoid default argument? This doesn't work in Java.
   
   That's why I recommend focusing on one language only and think of others later. For Java we might need more functions to look Scala-ish. No idea what R requires.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765516101


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38957/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767273142


   **[Test build #134474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134474/testReport)** for PR 31073 at commit [`6c0f6bb`](https://github.com/apache/spark/commit/6c0f6bbd7036851f27b0540fc687508342477c0c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553120084



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       Can we avoid default argument? This doesn't work in Java.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-760386251


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38647/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-757523412


   @yaooqinn - thanks for commenting.  It's generally easier to work with functions/classes that take Column arguments instead of Integer / Long arguments.
   
   Suppose you have a DataFrame with a `TimestampType` column called `some_time` and a `LongType` column called `some_millis`.  I don't think there is an easy way to build a `CalendarInterval` object from the `some_millis` column and add it to the `some_time` column.  This does not work:
   
   ```scala
   df.withColumn("new_time", col("d") + new CalendarInterval(0, 0, col("milliseconds")))
   ```
   
   Other datetime functions have also had this problem in the past.  In Spark 1.5 the `date_add` function was added with this method signature: `def date_add(start: Column, days: Int): Column`.  In Spark 3, this method signature was added to make `date_add` more usable: `def date_add(start: Column, days: Column): Column`, see [this JIRA](https://issues.apache.org/jira/browse/SPARK-28598).  It's easy to go from a Scala object to a Column object with `lit`.  It's not easy to go from a Column object to a Scala object.  
   
   `CalendarInterval` is also a bit more complicated for less technical users.  They need to run `import org.apache.spark.unsafe.types.CalendarInterval` and need to learn how to instantiate Java classes.  They also need to figure out how to convert hours / minutes to microseconds.  Think `CalendarInterval` is a great abstraction and is useful in a lot of contexts, but think `make_interval()` will be easier for most users.
   
   Let me know if any of these arguments convince you that `make_interval()` is a good idea ;)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765471117


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134370/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756420057


   **[Test build #133809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133809/testReport)** for PR 31073 at commit [`c9492a9`](https://github.com/apache/spark/commit/c9492a99ffa8a3ab1380ebde21c1ea200d02c187).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553431862



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       @HyukjinKwon - thank you for the code review.  The default arguments allow for this nice syntax:
   
   ```scala
   df.withColumn("plus_2_hours", col("first_datetime") + make_interval(hours = lit(2)))
   ```
   
   Without the default arguments, we'll need to invoke the method with a lot of arguments:
   
   ```scala
   val twoHourInterval = make_interval(lit(0), lit(0), lit(0), lit(0), lit(2), lit(0), lit(0))
   df.withColumn("plus_2_hours", col("first_datetime") + twoHourInterval)
   ```
   
   We could overload the method and allow for both syntaxes (using a little trick with the last argument that's optional).
   
   ```scala
   def make_interval(
       years: Column,
       months: Column,
       weeks: Column,
       days: Column,
       hours: Column,
       mins: Column,
       secs: Column,
       failOnError: Boolean): Column = withExpr {
     MakeInterval(
       years.expr, months.expr, weeks.expr, days.expr, hours.expr, mins.expr, secs.expr, failOnError)
   }
   ```
   
   This would allow Scala users to either manually list 8 arguments (e.g. `make_interval(lit(0), lit(0), lit(0), lit(0), lit(2), lit(0), lit(0), true)` or use the clean default syntax (e.g. `make_interval(hours = lit(2))`).  Would this give the Java users a usable interface?
   
   I'm also fine removing the default values if that's what's necessary to make this method compatible with Java.  @MaxGekk - open to your thoughts too cause I know you were originally in favor [of the default arguments approach](https://issues.apache.org/jira/browse/SPARK-33995?focusedCommentId=17258718&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17258718).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r558838730



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       LGTM @MrPowers 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756447462


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38398/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-757060723


   > @zero323 - There is a `make_interval` Postgres function and [make_interval is already part of the public SQL API](https://spark.apache.org/docs/latest/api/sql/index.html#make_interval). There are also `make_date` and `make_timestamp` SQL functions. There's a good argument to keep the function name as is for consistency.
   > 
   > I see your point and think `time_interval` is more descriptive. It seems like most of the datetime function names are just copied from Databases (e.g. `trunc` is from Oracle, `date_sub` is from MySQL, `datediff` is from SQL Server). I'd prefer using descriptive names to avoid API inconsistencies like having some method names using snake_case and others using all lowercase. But it seems like the precedent is to mimic the names used in databases.
   
   Makes sense. Thanks for clarification @MrPowers 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nvander1 commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
nvander1 commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-757366750


   For the Java users what about also adding something like an IntervalBuilder
   with methods for .years(...) .days(...)
   Then .build() will produce a Column ?
   
   Then they wouldn’t have to specify all 7 arguments and the user experiences
   would be similar to the default args the Scala users can benefit from.
   
   On Sat, Jan 9, 2021 at 2:14 PM Matthew Powers <no...@github.com>
   wrote:
   
   > *@MrPowers* commented on this pull request.
   > ------------------------------
   >
   > In sql/core/src/main/scala/org/apache/spark/sql/functions.scala
   > <https://github.com/apache/spark/pull/31073#discussion_r554461860>:
   >
   > > +  /**
   > +   * Creates a datetime interval
   > +   *
   > +   * @param years Number of years
   > +   * @param months Number of months
   > +   * @param weeks Number of weeks
   > +   * @param days Number of days
   > +   * @param hours Number of hours
   > +   * @param mins Number of mins
   > +   * @param secs Number of secs
   > +   * @return A datetime interval
   > +   * @group datetime_funcs
   > +   * @since 3.2.0
   > +   */
   > +  def make_interval(
   > +      years: Column = lit(0),
   >
   > @MaxGekk <https://github.com/MaxGekk> @HyukjinKwon
   > <https://github.com/HyukjinKwon> - Here's where we currently stand on the
   > default arguments issue:
   >
   >    - Default arguments are not used by other functions in
   >    org.apache.spark.sql.functions, so exposing a function with default
   >    param values isn't consistent with what's been done in the past.
   >    - The default arguments to make_interval make the function better for
   >    Scala users because it allows for this nice syntax make_interval(hours
   >    = lit(2))
   >    - Java users can use the make_interval function that's defined with
   >    the default arguments, as illustrated in this test
   >    <https://github.com/apache/spark/pull/31073/commits/c9492a99ffa8a3ab1380ebde21c1ea200d02c187>
   >    as long as they supply all seven arguments
   >
   > There are good arguments to use and to avoid default arguments with
   > make_interval. Can you please let me know your updated thoughts on if you
   > think we should use the default values or not? Thank you.
   >
   > —
   > You are receiving this because you commented.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/spark/pull/31073#discussion_r554461860>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ACXK4VZKAUNROENLLJOQYXDSZCTJBANCNFSM4VYECTKA>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765435887


   **[Test build #134370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134370/testReport)** for PR 31073 at commit [`0104388`](https://github.com/apache/spark/commit/01043882e3525d3f280b37bcf0d448b91ce7a46c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756420282


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133809/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553120021



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(

Review comment:
       Can we expose in SparkR and PySpark too? If you're not used to it, we can run it separately too. cc @zero323 FYI




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755939069


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38365/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-760357766


   **[Test build #134061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134061/testReport)** for PR 31073 at commit [`3b6953f`](https://github.com/apache/spark/commit/3b6953f0bbc91303dbecd7265675c017e6029205).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765484449


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38957/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r554457442



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(

Review comment:
       @zero323 - are you OK if I create a separate JIRA and do a separate pull request to add this function to PySpark & R?  Seems like @HyukjinKwon is open to doing everything in one PR or having separate PRs.  @jaceklaskowski prefers one PR for language.  Just let me know how you'd like me to proceed to keep this moving forward, thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r557556637



##########
File path: sql/core/src/test/java/test/org/apache/spark/sql/JavaDateFunctionsSuite.java
##########
@@ -0,0 +1,50 @@
+package test.org.apache.spark.sql;
+
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.test.TestSparkSession;
+import org.apache.spark.sql.types.StructType;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.Date;
+import java.util.*;
+
+import static org.apache.spark.sql.types.DataTypes.*;
+import static org.apache.spark.sql.functions.*;
+
+public class JavaDateFunctionsSuite {
+    private transient TestSparkSession spark;
+
+    @Before
+    public void setUp() {
+        spark = new TestSparkSession();
+    }
+
+    @After
+    public void tearDown() {
+        spark.stop();
+        spark = null;
+    }
+
+    @Test
+    public void makeIntervalWorksWithJava() {
+        Column twoYears = make_interval(lit(2), lit(0), lit(0), lit(0), lit(0), lit(0), lit(0));
+        List<Row> rows = Arrays.asList(
+                RowFactory.create(Date.valueOf("2014-06-30"), Date.valueOf("2016-06-30")),
+                RowFactory.create(Date.valueOf("2015-05-01"), Date.valueOf("2017-05-01")),
+                RowFactory.create(Date.valueOf("2018-12-30"), Date.valueOf("2020-12-30")));
+        StructType schema = createStructType(Arrays.asList(
+                createStructField("some_date", DateType, false),
+                createStructField("expected", DateType, false)));
+        Dataset<Row> df = spark.createDataFrame(rows, schema).withColumn("plus_two_years", col("some_date").plus(twoYears));
+        Assert.assertTrue(Arrays.equals(
+                (Row[]) df.select(df.col("plus_two_years")).collect(),
+                (Row[]) df.select(df.col("expected")).collect()));
+    }
+
+}

Review comment:
       Good catch, updated the spacing of this file.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755911521


   **[Test build #133776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133776/testReport)** for PR 31073 at commit [`39eac18`](https://github.com/apache/spark/commit/39eac182ac43dc4e77b982004c65195ff4b452ee).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756423010


   Wouldn't be better to call it just `interval` (or `time_interval` if we want to be specific)? `make_` seems a bit redundant here.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755976428


   **[Test build #133776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133776/testReport)** for PR 31073 at commit [`39eac18`](https://github.com/apache/spark/commit/39eac182ac43dc4e77b982004c65195ff4b452ee).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765516101


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38957/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767290683


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39060/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-760357817


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134061/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756420251


   **[Test build #133809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133809/testReport)** for PR 31073 at commit [`c9492a9`](https://github.com/apache/spark/commit/c9492a99ffa8a3ab1380ebde21c1ea200d02c187).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `public class JavaDateFunctionsSuite `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775326116


   **[Test build #135029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135029/testReport)** for PR 31073 at commit [`6c0f6bb`](https://github.com/apache/spark/commit/6c0f6bbd7036851f27b0540fc687508342477c0c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755743719


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765436423


   **[Test build #134370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134370/testReport)** for PR 31073 at commit [`0104388`](https://github.com/apache/spark/commit/01043882e3525d3f280b37bcf0d448b91ce7a46c).
    * This patch **fails RAT tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-760357438


   **[Test build #134061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134061/testReport)** for PR 31073 at commit [`3b6953f`](https://github.com/apache/spark/commit/3b6953f0bbc91303dbecd7265675c017e6029205).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756464867


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38398/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31073:
URL: https://github.com/apache/spark/pull/31073


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767290683


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39060/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-760431577


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38647/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jaceklaskowski commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
jaceklaskowski commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553969891



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(

Review comment:
       I recommend creating a JIRA issue for the other langs and keep focusing on one language only. That will make code review easier (for one-language-only eyes like mine).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-776412824


   thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r558776410



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       @zero323 - that's a good point.  I plan on creating additional PRs for some of the other functions that are in the SQL API, but aren't in the Scala/Python APIs yet and I'll just add functions for all the APIs in a single PR going forward.
   
   Do you have any additional feedback for this PR or think it's good to get merged?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r554513291



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(

Review comment:
       Yeah, we can run it separately.

##########
File path: sql/core/src/test/java/test/org/apache/spark/sql/JavaDateFunctionsSuite.java
##########
@@ -0,0 +1,50 @@
+package test.org.apache.spark.sql;
+
+import org.apache.spark.sql.Column;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.test.TestSparkSession;
+import org.apache.spark.sql.types.StructType;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.sql.Date;
+import java.util.*;
+
+import static org.apache.spark.sql.types.DataTypes.*;
+import static org.apache.spark.sql.functions.*;
+
+public class JavaDateFunctionsSuite {
+    private transient TestSparkSession spark;
+
+    @Before
+    public void setUp() {
+        spark = new TestSparkSession();
+    }
+
+    @After
+    public void tearDown() {
+        spark.stop();
+        spark = null;
+    }
+
+    @Test
+    public void makeIntervalWorksWithJava() {
+        Column twoYears = make_interval(lit(2), lit(0), lit(0), lit(0), lit(0), lit(0), lit(0));
+        List<Row> rows = Arrays.asList(
+                RowFactory.create(Date.valueOf("2014-06-30"), Date.valueOf("2016-06-30")),
+                RowFactory.create(Date.valueOf("2015-05-01"), Date.valueOf("2017-05-01")),
+                RowFactory.create(Date.valueOf("2018-12-30"), Date.valueOf("2020-12-30")));
+        StructType schema = createStructType(Arrays.asList(
+                createStructField("some_date", DateType, false),
+                createStructField("expected", DateType, false)));
+        Dataset<Row> df = spark.createDataFrame(rows, schema).withColumn("plus_two_years", col("some_date").plus(twoYears));
+        Assert.assertTrue(Arrays.equals(
+                (Row[]) df.select(df.col("plus_two_years")).collect(),
+                (Row[]) df.select(df.col("expected")).collect()));
+    }
+
+}

Review comment:
       Can we use 2-space indentation?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-757419515


   @cloud-fan @yaooqinn WDYT?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-760357438


   **[Test build #134061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134061/testReport)** for PR 31073 at commit [`3b6953f`](https://github.com/apache/spark/commit/3b6953f0bbc91303dbecd7265675c017e6029205).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775344056


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135029/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767383031


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134474/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755911521


   **[Test build #133776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133776/testReport)** for PR 31073 at commit [`39eac18`](https://github.com/apache/spark/commit/39eac182ac43dc4e77b982004c65195ff4b452ee).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r554514614



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),
+      months: Column = lit(0),
+      weeks: Column = lit(0),
+      days: Column = lit(0),
+      hours: Column = lit(0),
+      mins: Column = lit(0),
+      secs: Column = lit(0)): Column = withExpr {
+    MakeInterval(years.expr, months.expr, weeks.expr, days.expr, hours.expr, mins.expr, secs.expr)
+  }

Review comment:
       Can we just match how it's invoked in SQL side? We can have 7 overridden versions or leverage `varargs` like https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L493-L494




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755743719


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775120028


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775188087


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39614/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755957486


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38365/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
zero323 commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r558616172



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       > That's why I recommend focusing on one language only and think of others later. For Java we might need more functions to look Scala-ish. No idea what R requires.
   
   While I don't see a huge issue with adding Python and R API separately for this specific function, Scala API often sets limits for what is possible on the other side.  Moreover, we seldom extend JVM API just for the sake of guest languages these days. So unless API is trivial (like here) or uses well established interface (like ML model), it's better to design things at the same time.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765435887


   **[Test build #134370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134370/testReport)** for PR 31073 at commit [`0104388`](https://github.com/apache/spark/commit/01043882e3525d3f280b37bcf0d448b91ce7a46c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756464867


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38398/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755986593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133776/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775200147


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39614/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767290666


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39060/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-759461647


   I think it's a good point that the nice usage `make_interval(hour = lit(1))` is not possible in Java. This is similar to `Dataset` APIs while Scala users can rely on implicit but Java users must specify the encoder. I'm OK with it if there is no better way, and Java users can still use this API, just less convenient.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-760404103


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38647/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553619800



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       @MaxGekk - I added a test to demonstrate that `make_interval` can be called from the Java API (with the Scala method that has default values) when all 7 arguments are passed to the method.  e.g. this Java code works:
   
   ```java
   Column twoYears = make_interval(lit(2), lit(0), lit(0), lit(0), lit(0), lit(0), lit(0));
   Dataset<Row> df = spark.createDataFrame(rows, schema).withColumn("plus_two_years", col("some_date").plus(twoYears));
   ```
   
   Someone with Java experience should check my result.  If the current implementation works with the Java API, then I think it's ok to keep it, especially because it allows for much more elegant Scala code.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765461483


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38957/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775344056


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135029/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767372131


   **[Test build #134474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134474/testReport)** for PR 31073 at commit [`6c0f6bb`](https://github.com/apache/spark/commit/6c0f6bbd7036851f27b0540fc687508342477c0c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756437075


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38398/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r554457442



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(

Review comment:
       @zero323 - are you OK if I create a separate JIRA and do a separate pull request to add this function to PySpark & R?  Seems like @HyukjinKwon is open to doing everything in one PR or having separate PRs.  @jaceklaskowski prefers one PR for language.  Just let me know how you'd like me to proceed to keep this moving forward, thanks!

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       @MaxGekk @HyukjinKwon - Here's where we currently stand on the default arguments issue:
   
   * Default arguments are not used by other functions in `org.apache.spark.sql.functions`, so exposing a function with default param values isn't consistent with what's been done in the past.
   * The default arguments to `make_interval` make the function better for Scala users because it allows for this nice syntax `make_interval(hours = lit(2))`
   * Java users can use the `make_interval` function that's defined with the default arguments, [as illustrated in this test](https://github.com/apache/spark/pull/31073/commits/c9492a99ffa8a3ab1380ebde21c1ea200d02c187) as long as they supply all seven arguments
   
   There are good arguments to use and to avoid default arguments with `make_interval`.  Can you please let me know your updated thoughts on if you think we should use the default values or not?  Thank you.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jaceklaskowski commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
jaceklaskowski commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553971028



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       > Can we avoid default argument? This doesn't work in Java.
   
   That's why I recommend focusing on one language only and think of others later. For Java we might need more functions to look Scala-ish. No idea what R requires.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] zero323 commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
zero323 commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-757060723


   > @zero323 - There is a `make_interval` Postgres function and [make_interval is already part of the public SQL API](https://spark.apache.org/docs/latest/api/sql/index.html#make_interval). There are also `make_date` and `make_timestamp` SQL functions. There's a good argument to keep the function name as is for consistency.
   > 
   > I see your point and think `time_interval` is more descriptive. It seems like most of the datetime function names are just copied from Databases (e.g. `trunc` is from Oracle, `date_sub` is from MySQL, `datediff` is from SQL Server). I'd prefer using descriptive names to avoid API inconsistencies like having some method names using snake_case and others using all lowercase. But it seems like the precedent is to mimic the names used in databases.
   
   Makes sense. Thanks for clarification @MrPowers 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r557557211



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval

Review comment:
       Sure, added `(Scala-specific)`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-760357817






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nvander1 commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
nvander1 commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-757366750


   For the Java users what about also adding something like an IntervalBuilder
   with methods for .years(...) .days(...)
   Then .build() will produce a Column ?
   
   Then they wouldn’t have to specify all 7 arguments and the user experiences
   would be similar to the default args the Scala users can benefit from.
   
   On Sat, Jan 9, 2021 at 2:14 PM Matthew Powers <no...@github.com>
   wrote:
   
   > *@MrPowers* commented on this pull request.
   > ------------------------------
   >
   > In sql/core/src/main/scala/org/apache/spark/sql/functions.scala
   > <https://github.com/apache/spark/pull/31073#discussion_r554461860>:
   >
   > > +  /**
   > +   * Creates a datetime interval
   > +   *
   > +   * @param years Number of years
   > +   * @param months Number of months
   > +   * @param weeks Number of weeks
   > +   * @param days Number of days
   > +   * @param hours Number of hours
   > +   * @param mins Number of mins
   > +   * @param secs Number of secs
   > +   * @return A datetime interval
   > +   * @group datetime_funcs
   > +   * @since 3.2.0
   > +   */
   > +  def make_interval(
   > +      years: Column = lit(0),
   >
   > @MaxGekk <https://github.com/MaxGekk> @HyukjinKwon
   > <https://github.com/HyukjinKwon> - Here's where we currently stand on the
   > default arguments issue:
   >
   >    - Default arguments are not used by other functions in
   >    org.apache.spark.sql.functions, so exposing a function with default
   >    param values isn't consistent with what's been done in the past.
   >    - The default arguments to make_interval make the function better for
   >    Scala users because it allows for this nice syntax make_interval(hours
   >    = lit(2))
   >    - Java users can use the make_interval function that's defined with
   >    the default arguments, as illustrated in this test
   >    <https://github.com/apache/spark/pull/31073/commits/c9492a99ffa8a3ab1380ebde21c1ea200d02c187>
   >    as long as they supply all seven arguments
   >
   > There are good arguments to use and to avoid default arguments with
   > make_interval. Can you please let me know your updated thoughts on if you
   > think we should use the default values or not? Thank you.
   >
   > —
   > You are receiving this because you commented.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/spark/pull/31073#discussion_r554461860>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ACXK4VZKAUNROENLLJOQYXDSZCTJBANCNFSM4VYECTKA>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756446704


   @zero323 - There is a `make_interval` Postgres function and [make_interval is already part of the public SQL API](https://spark.apache.org/docs/latest/api/sql/index.html#make_interval).  There are also `make_date` and `make_timestamp` SQL functions.  There's a good argument to keep the function name as is for consistency.
   
   I see your point and think `time_interval` is more descriptive.  It seems like most of the datetime function names are just copied from Databases (e.g. `trunc` is from Oracle, `date_sub` is from MySQL, `datediff` is from SQL Server).  I'd prefer using descriptive names to avoid API inconsistencies like having some method names using snake_case and others using all lowercase.  But it seems like the precedent is to mimic the names used in databases.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775144967


   **[Test build #135029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135029/testReport)** for PR 31073 at commit [`6c0f6bb`](https://github.com/apache/spark/commit/6c0f6bbd7036851f27b0540fc687508342477c0c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553469383



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       What kind of issue will we face? Do we have to specify all arguments when we call such method from Java, or impossible to call it all?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       What kind of issue will we face to? Do we have to specify all arguments when we call such method from Java, or impossible to call it all?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r554514044



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval

Review comment:
       Can we add `(Scala-specific) ` here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767273142


   **[Test build #134474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134474/testReport)** for PR 31073 at commit [`6c0f6bb`](https://github.com/apache/spark/commit/6c0f6bbd7036851f27b0540fc687508342477c0c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775221212


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39614/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755957486


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38365/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-765471117


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134370/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775221212


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39614/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] yaooqinn commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-757466168


   For me, to get an interval at the scala/java side, the `lit` and `CalendarInterval` is much easier and clearer.
   ```scala
   lit(new CalendarInterval(0, 1, 0))
   res0: org.apache.spark.sql.Column = INTERVAL '1 days'
   ```
   
   So I am sitting at the fence for this change.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-774492810


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756420282


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133809/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553619800



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       @MaxGekk - I [added a test](https://github.com/apache/spark/pull/31073/commits/c9492a99ffa8a3ab1380ebde21c1ea200d02c187) to demonstrate that `make_interval` can be called from the Java API (with the Scala method that has default values) when all 7 arguments are passed to the method.  e.g. this Java code works:
   
   ```java
   Column twoYears = make_interval(lit(2), lit(0), lit(0), lit(0), lit(0), lit(0), lit(0));
   Dataset<Row> df = spark.createDataFrame(rows, schema).withColumn("plus_two_years", col("some_date").plus(twoYears));
   ```
   
   Someone with Java experience should check my result.  If the current implementation works with the Java API, then I think it's ok to keep it, especially because it allows for much more elegant Scala code.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-756420057


   **[Test build #133809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133809/testReport)** for PR 31073 at commit [`c9492a9`](https://github.com/apache/spark/commit/c9492a99ffa8a3ab1380ebde21c1ea200d02c187).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-775144967


   **[Test build #135029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135029/testReport)** for PR 31073 at commit [`6c0f6bb`](https://github.com/apache/spark/commit/6c0f6bbd7036851f27b0540fc687508342477c0c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767273142






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-774492810


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-767383031


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134474/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755897749






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31073:
URL: https://github.com/apache/spark/pull/31073#issuecomment-755927168


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38365/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r553619800



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       @MaxGekk - I [added a test](https://github.com/apache/spark/pull/31073/commits/c9492a99ffa8a3ab1380ebde21c1ea200d02c187) to demonstrate that `make_interval` can be called from the Java API (with the Scala method that has default values) when all 7 arguments are passed to the method.  e.g. this Java code works:
   
   ```java
   Column twoYears = make_interval(lit(2), lit(0), lit(0), lit(0), lit(0), lit(0), lit(0));
   Dataset<Row> df = spark.createDataFrame(rows, schema).withColumn("plus_two_years", col("some_date").plus(twoYears));
   ```
   
   Someone with Java experience should check my result.  If the current implementation with default parameters works with the Java API, then I think it's ok to keep it, especially because it allows for much more elegant Scala code.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MrPowers commented on a change in pull request #31073: [SPARK-33995][SQL] Expose make_interval as a Scala function

Posted by GitBox <gi...@apache.org>.
MrPowers commented on a change in pull request #31073:
URL: https://github.com/apache/spark/pull/31073#discussion_r554461860



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2841,6 +2841,31 @@ object functions {
   // DateTime functions
   //////////////////////////////////////////////////////////////////////////////////////////////
 
+  /**
+   * Creates a datetime interval
+   *
+   * @param years Number of years
+   * @param months Number of months
+   * @param weeks Number of weeks
+   * @param days Number of days
+   * @param hours Number of hours
+   * @param mins Number of mins
+   * @param secs Number of secs
+   * @return A datetime interval
+   * @group datetime_funcs
+   * @since 3.2.0
+   */
+  def make_interval(
+      years: Column = lit(0),

Review comment:
       @MaxGekk @HyukjinKwon - Here's where we currently stand on the default arguments issue:
   
   * Default arguments are not used by other functions in `org.apache.spark.sql.functions`, so exposing a function with default param values isn't consistent with what's been done in the past.
   * The default arguments to `make_interval` make the function better for Scala users because it allows for this nice syntax `make_interval(hours = lit(2))`
   * Java users can use the `make_interval` function that's defined with the default arguments, [as illustrated in this test](https://github.com/apache/spark/pull/31073/commits/c9492a99ffa8a3ab1380ebde21c1ea200d02c187) as long as they supply all seven arguments
   
   There are good arguments to use and to avoid default arguments with `make_interval`.  Can you please let me know your updated thoughts on if you think we should use the default values or not?  Thank you.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org