You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by youngbink <gi...@git.apache.org> on 2017/12/19 01:55:14 UTC

[GitHub] spark pull request #20015: Add new built-in function date_trunc()

GitHub user youngbink opened a pull request:

    https://github.com/apache/spark/pull/20015

    Add new built-in function date_trunc()

    ## What changes were proposed in this pull request?
    
    Adding date_trunc() as a built-in function.
    date_trunc() is common in other databases, but Spark or Hive does not have support for this.
    We do have trunc() but this only works with 'MONTH' and 'YEAR' level on the DateType input.
    
    date_trunc() in other databases:
    AWS Redshift: http://docs.aws.amazon.com/redshift/latest/dg/r_DATE_TRUNC.html
    PostgreSQL: https://www.postgresql.org/docs/9.1/static/functions-datetime.html
    Presto: https://prestodb.io/docs/current/functions/datetime.html
    
    
    ## How was this patch tested?
    
    Unit tests
    
    (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
    (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/youngbink/spark date_trunc

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20015.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20015
    
----
commit d84fc0ef0d605ba8944de64d3f6e0ac3d0da5a1b
Author: Youngbin Kim <yk...@hotmail.com>
Date:   2017-12-19T01:36:29Z

    date_trunc support

commit f94f401bcfd765b21c3fb466041b42a605d6a814
Author: Youngbin Kim <yk...@hotmail.com>
Date:   2017-12-19T01:53:43Z

    minor edits: version

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157908653
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,183 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   *
    --- End diff --
    
    Remove this line.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85131 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85131/testReport)** for PR 20015 at commit [`3547b7c`](https://github.com/apache/spark/commit/3547b7c0bf8caa972e018f6875e4b0f599c4e12f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157675835
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,184 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncTime extends BinaryExpression with ImplicitCastInputTypes {
    --- End diff --
    
    Maybe `TruncInstant`? I received this advice before and I liked it too. Not a big deal tho.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: Add new built-in function date_trunc()

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    ok to test


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157673559
  
    --- Diff: python/pyspark/sql/functions.py ---
    @@ -1111,6 +1111,24 @@ def trunc(date, format):
         return Column(sc._jvm.functions.trunc(_to_java_column(date), format))
     
     
    +@since(2.3)
    +def date_trunc(format, timestamp):
    +    """
    +    Returns timestamp truncated to the unit specified by the format.
    +
    +    :param format: 'year', 'YYYY', 'yy', 'month', 'mon', 'mm',
    +        'DAY', 'DD', 'HOUR', 'MINUTE', 'SECOND', 'WEEK', 'QUARTER'
    +
    +    >>> df = spark.createDataFrame([('1997-02-28',)], ['d'])
    --- End diff --
    
    Can we use a timestamp string like `1997-02-28 05:02:11` to show the difference from `trunc` a bit more clearly?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85083/testReport)** for PR 20015 at commit [`f94f401`](https://github.com/apache/spark/commit/f94f401bcfd765b21c3fb466041b42a605d6a814).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85083/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157686437
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,184 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncTime extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   *
    +   * @param input
    +   * @param maxLevel Maximum level that can be used for truncation (e.g MONTH for Date input)
    +   * @param truncFunc
    +   * @tparam T
    +   * @return
    +   */
    +  protected def evalHelper[T](input: InternalRow, maxLevel: Int)(
    +    truncFunc: (Any, Int) => T): Any = {
    --- End diff --
    
    Maybe `truncFunc: (Any, Int) => Any` is enough? So we don't need to use the `T`, but I'm not sure if this is better... 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157926055
  
    --- Diff: python/pyspark/sql/functions.py ---
    @@ -1111,6 +1111,24 @@ def trunc(date, format):
         return Column(sc._jvm.functions.trunc(_to_java_column(date), format))
     
     
    +@since(2.3)
    +def date_trunc(format, timestamp):
    +    """
    +    Returns timestamp truncated to the unit specified by the format.
    +
    +    :param format: 'year', 'yyyy', 'yy', 'month', 'mon', 'mm',
    +        'day', 'dd', 'hour', 'minute', 'second', 'week', 'quarter'
    +
    +    >>> df = spark.createDataFrame([('1997-02-28 05:02:11',)], ['d'])
    --- End diff --
    
    `d` -> `t` or `ts`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85131/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85141/testReport)** for PR 20015 at commit [`0d1a8cb`](https://github.com/apache/spark/commit/0d1a8cbc922bc410d8d4a69c26b16290773a197c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157677400
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
    @@ -2797,6 +2797,21 @@ object functions {
         TruncDate(date.expr, Literal(format))
       }
     
    +  /**
    +   * Returns timestamp truncated to the unit specified by the format.
    +   *
    +   * @param format: 'year', 'yyyy', 'yy' for truncate by year,
    +   *               'month', 'mon', 'mm' for truncate by month,
    +   *               'day', 'dd' for truncate by day,
    +   *               Other options are: second, minute, hour, week, month, quarter
    --- End diff --
    
    Maybe, `'second', 'minute', 'hour', 'week', 'month' and 'quarter'`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157677136
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
    @@ -2797,6 +2797,21 @@ object functions {
         TruncDate(date.expr, Literal(format))
       }
     
    +  /**
    +   * Returns timestamp truncated to the unit specified by the format.
    +   *
    +   * @param format: 'year', 'yyyy', 'yy' for truncate by year,
    +   *               'month', 'mon', 'mm' for truncate by month,
    --- End diff --
    
    nit: one space each more.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by youngbink <gi...@git.apache.org>.
Github user youngbink commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    @HyukjinKwon  Just took a look at this PR #14788. 
    
    My point of mentioning those databases was just to give examples of the function that Spark doesn't support but other databases commonly do. (They all have this `date_trunc` which takes `timestamp` and output `timestamp`)
    As you said, we could extend `trunc` and simply create an alias `date_trunc`, but it's actually not as simple. For e.g, PR #14788 won't be able to handle the following command collectly on PySpark:
    ```
    df = spark.createDataFrame([('1997-02-28 05:02:11',)], ['d'])
    df.select(functions.trunc(df.d, 'year').alias('year')).collect()  
    df.select(functions.trunc(df.d, 'SS').alias('SS')).collect() 
    ```
    This is because `trunc(string, string)` isn't correctly handled. We could find a way around this and get it working, but after having a discussion with @cloud-fan, @gatorsmile, @rednaxelafx and Reynold, we decided to add `date_trunc` to be compatible with Postgres for now. 
    
    
    
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157680290
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala ---
    @@ -944,9 +954,16 @@ object DateTimeUtils {
         date + daysToMonthEnd
       }
     
    -  private val TRUNC_TO_YEAR = 1
    -  private val TRUNC_TO_MONTH = 2
    -  private val TRUNC_INVALID = -1
    +  // Visible for testing.
    +  val TRUNC_TO_YEAR = 1
    +  val TRUNC_TO_MONTH = 2
    +  val TRUNC_TO_DAY = 3
    +  val TRUNC_TO_HOUR = 4
    +  val TRUNC_TO_MINUTE = 5
    +  val TRUNC_TO_SECOND = 6
    +  val TRUNC_TO_WEEK = 7
    +  val TRUNC_TO_QUARTER = 8
    +  val TRUNC_INVALID = -1
    --- End diff --
    
    Can we bring quarter and week forward, maybe to 3 and 4? Then it's more conform to the order of time granularity and max-level design is not influenced.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20015


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85140/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    @cloud-fan and @youngbink how about reviving https://github.com/apache/spark/pull/14788 with a configuration to control this?
    
    AWS Redshift seems having `TRUNC` which just converts a timestamp to a date whereas we have Spark's `trunc` where supports date formats. This is not quite equivalent. I think Spark's `trunc` is more like Redshift's `DATE_TRUNC`.
    
    PostgreSQL does not have `trunc` but has `date_trunc` where we can specify the format and returns a timestamp always.
    
    Presto also looks not having a duplicated functionality.
    
    I think we can simply introduce an alias for `trunc` after resolving https://github.com/apache/spark/pull/14788.
    
    Did I maybe miss something?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    If we haven't get a similar function, I would have gone +1 but what I am less sure is `date_trunc` actually quite sounds a better version of `trunc` to be honest. Seems both also extend the same parent here `TruncTime`.
    
    I feel like we are trying to add this better version alone by working around because it takes a relatively larger change to update other related functions consistently.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85146/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157676669
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,184 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncTime extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   *
    +   * @param input
    +   * @param maxLevel Maximum level that can be used for truncation (e.g MONTH for Date input)
    +   * @param truncFunc
    +   * @tparam T
    +   * @return
    +   */
    +  protected def evalHelper[T](input: InternalRow, maxLevel: Int)(
    +    truncFunc: (Any, Int) => T): Any = {
         val level = if (format.foldable) {
           truncLevel
         } else {
           DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
         }
    -    if (level == -1) {
    +    if (level == DateTimeUtils.TRUNC_INVALID || level > maxLevel) {
    --- End diff --
    
    `// unknown format or too small level`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157677311
  
    --- Diff: python/pyspark/sql/functions.py ---
    @@ -1111,6 +1111,24 @@ def trunc(date, format):
         return Column(sc._jvm.functions.trunc(_to_java_column(date), format))
     
     
    +@since(2.3)
    +def date_trunc(format, timestamp):
    +    """
    +    Returns timestamp truncated to the unit specified by the format.
    +
    +    :param format: 'year', 'YYYY', 'yy', 'month', 'mon', 'mm',
    +        'DAY', 'DD', 'HOUR', 'MINUTE', 'SECOND', 'WEEK', 'QUARTER'
    --- End diff --
    
    Could we make those lowercased too?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: Add new built-in function date_trunc()

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    we need to create a JIRA ticket


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by youngbink <gi...@git.apache.org>.
Github user youngbink commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    hmm...even if we decide to change this later, I honestly think merging `trunc` and `date_trunc` would be a simple work that only touches a couple of files, mostly in `datetimeExpressions.scala`.
    This PR isn't too small as you said, but most of the codes here can be used without modification when we merge `date_trunc`. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157896273
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala ---
    @@ -964,7 +981,62 @@ object DateTimeUtils {
       }
     
       /**
    -   * Returns the truncate level, could be TRUNC_YEAR, TRUNC_MONTH, or TRUNC_INVALID,
    +   * Returns the trunc date time from original date time and trunc level.
    +   * Trunc level should be generated using `parseTruncLevel()`, should be between 1 and 8
    +   */
    +  def truncTimestamp(d: SQLTimestamp, level: Int, timeZone: TimeZone): SQLTimestamp = {
    --- End diff --
    
    nit: d -> ts or t


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    SPARK-17174 originally described few functions related with hour, min, etc. but I received an advice to fix up other related functions too even though they could also be done alone too. I agreed with doing other functions too at that time and I tried to propose as so.
    
    I am saying I think this PR actually more targets adding another (better) version of `trunc` to support day, hour, min, etc. in the format. In this case, I think we should deduplicate the logics with related functions too.
    
    Ah, so, I think I am less sure about why this should be done alone leaving out other related changes, and other functions we (I) usually reject.
    
    and I think you and @cloud-fan say the reasons are, it's common and this PR targets a separate functionality consistent with other DBMS.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85131 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85131/testReport)** for PR 20015 at commit [`3547b7c`](https://github.com/apache/spark/commit/3547b7c0bf8caa972e018f6875e4b0f599c4e12f).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: Add new built-in function date_trunc()

Posted by youngbink <gi...@git.apache.org>.
Github user youngbink commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    @gatorsmile @cloud-fan 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157673626
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,184 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncTime extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   *
    +   * @param input
    --- End diff --
    
    Seems `input` and `truncFunc` descriptions missing.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157908622
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,183 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   *
    +   * @param input internalRow (time)
    +   * @param maxLevel Maximum level that can be used for truncation (e.g MONTH for Date input)
    +   * @param truncFunc function: (time, level) => time
    +   * @return
    --- End diff --
    
    Remove `@return`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157821540
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,184 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncTime extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   *
    +   * @param input
    +   * @param maxLevel Maximum level that can be used for truncation (e.g MONTH for Date input)
    +   * @param truncFunc
    +   * @tparam T
    +   * @return
    +   */
    +  protected def evalHelper[T](input: InternalRow, maxLevel: Int)(
    +    truncFunc: (Any, Int) => T): Any = {
         val level = if (format.foldable) {
           truncLevel
         } else {
           DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
         }
    -    if (level == -1) {
    +    if (level == DateTimeUtils.TRUNC_INVALID || level > maxLevel) {
           // unknown format
           null
         } else {
    -      val d = date.eval(input)
    +      val d = time.eval(input)
           if (d == null) {
             null
           } else {
    -        DateTimeUtils.truncDate(d.asInstanceOf[Int], level)
    +        truncFunc(d, level)
           }
         }
       }
     
    -  override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
    +  protected def codeGenHelper[T](
    +      ctx: CodegenContext,
    +      ev: ExprCode,
    +      maxLevel: Int,
    +      orderReversed: Boolean = false)(
    +      truncFunc: (String, String) => String)
    +    : ExprCode = {
         val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
     
         if (format.foldable) {
    -      if (truncLevel == -1) {
    +      if (truncLevel == DateTimeUtils.TRUNC_INVALID || truncLevel > maxLevel) {
             ev.copy(code = s"""
               boolean ${ev.isNull} = true;
               ${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};""")
           } else {
    -        val d = date.genCode(ctx)
    +        val d = time.genCode(ctx)
    +        val truncFuncStr = truncFunc(d.value, truncLevel.toString)
             ev.copy(code = s"""
               ${d.code}
               boolean ${ev.isNull} = ${d.isNull};
               ${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};
               if (!${ev.isNull}) {
    -            ${ev.value} = $dtu.truncDate(${d.value}, $truncLevel);
    +            ${ev.value} = $dtu.$truncFuncStr;
               }""")
           }
         } else {
    -      nullSafeCodeGen(ctx, ev, (dateVal, fmt) => {
    +      nullSafeCodeGen(ctx, ev, (left, right) => {
             val form = ctx.freshName("form")
    +        val (dateVal, fmt) = if (orderReversed) {
    +          (right, left)
    +        } else {
    +          (left, right)
    +        }
    +        val truncFuncStr = truncFunc(dateVal, form)
             s"""
               int $form = $dtu.parseTruncLevel($fmt);
    -          if ($form == -1) {
    +          if ($form == -1 || $form > $maxLevel) {
                 ${ev.isNull} = true;
               } else {
    -            ${ev.value} = $dtu.truncDate($dateVal, $form);
    +            ${ev.value} = $dtu.$truncFuncStr
               }
             """
           })
         }
       }
     }
     
    +/**
    + * Returns date truncated to the unit specified by the format.
    + */
    +// scalastyle:off line.size.limit
    +@ExpressionDescription(
    +  usage = """
    +    _FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.
    +    `fmt` should be one of ["YEAR", "YYYY", "YY", "MON", "MONTH", "MM"]
    --- End diff --
    
    Let us use the lower case and also update the other functions in this file. For example, `ToUnixTimestamp`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85132/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157674840
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,184 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncTime extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   *
    +   * @param input
    +   * @param maxLevel Maximum level that can be used for truncation (e.g MONTH for Date input)
    +   * @param truncFunc
    +   * @tparam T
    +   * @return
    +   */
    +  protected def evalHelper[T](input: InternalRow, maxLevel: Int)(
    +    truncFunc: (Any, Int) => T): Any = {
         val level = if (format.foldable) {
           truncLevel
         } else {
           DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
         }
    -    if (level == -1) {
    +    if (level == DateTimeUtils.TRUNC_INVALID || level > maxLevel) {
           // unknown format
           null
         } else {
    -      val d = date.eval(input)
    +      val d = time.eval(input)
    --- End diff --
    
    nit: Since this is a time, it can be `val t = ...`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85140/testReport)** for PR 20015 at commit [`80a1959`](https://github.com/apache/spark/commit/80a195989bd138fade5910cbf495b472a57ce445).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157905770
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,183 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    --- End diff --
    
    Maybe, `time` -> `instant`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85132 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85132/testReport)** for PR 20015 at commit [`b12ba92`](https://github.com/apache/spark/commit/b12ba92add942c087dd45933464937479fc24bcd).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85140/testReport)** for PR 20015 at commit [`80a1959`](https://github.com/apache/spark/commit/80a195989bd138fade5910cbf495b472a57ce445).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85146/testReport)** for PR 20015 at commit [`238d7d4`](https://github.com/apache/spark/commit/238d7d470c583c910bccbca8bbcaa681b67d6025).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85141/testReport)** for PR 20015 at commit [`0d1a8cb`](https://github.com/apache/spark/commit/0d1a8cbc922bc410d8d4a69c26b16290773a197c).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Yeah keep any substantive discussion on the public lists. Sometimes a side conversation happens; summarize the points here.
    
    We've rejected a lot of other functions that other DBs, but not Hive, support. Spark mostly follows Hive, and for everything else, there are UDFs. I'm not against this so much as not clear why it's exceptional


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    - The API proposed by this PR is consistent with the other DBs. 
    - The implementation does not introduce the behavior changes. 
    
    The implementation is clean and the PR quality is pretty good. 



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157925994
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,181 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes {
    +  val instant: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   * @param input internalRow (time)
    +   * @param maxLevel Maximum level that can be used for truncation (e.g MONTH for Date input)
    +   * @param truncFunc function: (time, level) => time
    +   */
    +  protected def evalHelper(input: InternalRow, maxLevel: Int)(
    +    truncFunc: (Any, Int) => Any): Any = {
         val level = if (format.foldable) {
           truncLevel
         } else {
           DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
         }
    -    if (level == -1) {
    -      // unknown format
    +    if (level == DateTimeUtils.TRUNC_INVALID || level > maxLevel) {
    +      // unknown format or too large level
           null
         } else {
    -      val d = date.eval(input)
    -      if (d == null) {
    +      val t = instant.eval(input)
    +      if (t == null) {
             null
           } else {
    -        DateTimeUtils.truncDate(d.asInstanceOf[Int], level)
    +        truncFunc(t, level)
           }
         }
       }
     
    -  override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
    +  protected def codeGenHelper(
    +      ctx: CodegenContext,
    +      ev: ExprCode,
    +      maxLevel: Int,
    +      orderReversed: Boolean = false)(
    +      truncFunc: (String, String) => String)
    +    : ExprCode = {
         val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
     
         if (format.foldable) {
    -      if (truncLevel == -1) {
    +      if (truncLevel == DateTimeUtils.TRUNC_INVALID || truncLevel > maxLevel) {
             ev.copy(code = s"""
               boolean ${ev.isNull} = true;
               ${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};""")
           } else {
    -        val d = date.genCode(ctx)
    +        val t = instant.genCode(ctx)
    +        val truncFuncStr = truncFunc(t.value, truncLevel.toString)
             ev.copy(code = s"""
    -          ${d.code}
    -          boolean ${ev.isNull} = ${d.isNull};
    +          ${t.code}
    +          boolean ${ev.isNull} = ${t.isNull};
               ${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};
               if (!${ev.isNull}) {
    -            ${ev.value} = $dtu.truncDate(${d.value}, $truncLevel);
    +            ${ev.value} = $dtu.$truncFuncStr;
               }""")
           }
         } else {
    -      nullSafeCodeGen(ctx, ev, (dateVal, fmt) => {
    +      nullSafeCodeGen(ctx, ev, (left, right) => {
             val form = ctx.freshName("form")
    +        val (dateVal, fmt) = if (orderReversed) {
    +          (right, left)
    +        } else {
    +          (left, right)
    +        }
    +        val truncFuncStr = truncFunc(dateVal, form)
             s"""
               int $form = $dtu.parseTruncLevel($fmt);
    -          if ($form == -1) {
    +          if ($form == -1 || $form > $maxLevel) {
                 ${ev.isNull} = true;
               } else {
    -            ${ev.value} = $dtu.truncDate($dateVal, $form);
    +            ${ev.value} = $dtu.$truncFuncStr
               }
             """
           })
         }
       }
     }
     
    +/**
    + * Returns date truncated to the unit specified by the format.
    + */
    +// scalastyle:off line.size.limit
    +@ExpressionDescription(
    +  usage = """
    +    _FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.
    +    `fmt` should be one of ["year", "yyyy", "yy", "mon", "month", "mm"]
    +  """,
    +  examples = """
    +    Examples:
    +      > SELECT _FUNC_('2009-02-12', 'MM');
    +       2009-02-01
    +      > SELECT _FUNC_('2015-10-27', 'YEAR');
    +       2015-01-01
    +  """,
    +  since = "1.5.0")
    +// scalastyle:on line.size.limit
    +case class TruncDate(date: Expression, format: Expression)
    +  extends TruncInstant {
    +  override def left: Expression = date
    +  override def right: Expression = format
    +
    +  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    +  override def dataType: DataType = DateType
    +  override def prettyName: String = "trunc"
    +  override val instant = date
    +
    +  override def eval(input: InternalRow): Any = {
    +    evalHelper(input, maxLevel = DateTimeUtils.TRUNC_TO_MONTH) { (d: Any, level: Int) =>
    +      DateTimeUtils.truncDate(d.asInstanceOf[Int], level)
    +    }
    +  }
    +
    +  override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
    +    codeGenHelper(ctx, ev, maxLevel = DateTimeUtils.TRUNC_TO_MONTH) { (date: String, fmt: String) =>
    +      s"truncDate($date, $fmt);"
    +    }
    +  }
    +}
    +
    +/**
    + * Returns timestamp truncated to the unit specified by the format.
    + */
    +// scalastyle:off line.size.limit
    +@ExpressionDescription(
    +  usage = """
    +    _FUNC_(fmt, date) - Returns timestamp `ts` truncated to the unit specified by the format model `fmt`.
    --- End diff --
    
    `date` -> `ts`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    I get `date_trunc` is common in other DBMS. I can see that this can be done now and we can still proceed `trunc`, etc. later. So, I am fine but still less sure tho.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85145/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    We had an offline discussion and wanna send this out to get more feedbacks. So generally just adding `date_trunc` is pretty straightforward and makes Spark consistent with other databases about this function, while extending `trunc` to support timestamp type is a better API design.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    > after having a discussion with @cloud-fan, @gatorsmile, @rednaxelafx and Reynold
    
    Where did the discussion happen? Was this offline discussion? I also want to actively join in the discussion. Many implementations of the trunc works differently and I think we decide the current behaviour after sufficient discussion.
    
    If we don't fix the stuff about #14788 in 2.3.0 timeline, it could be even more difficult because we need to keep the previous behaviour.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85083 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85083/testReport)** for PR 20015 at commit [`f94f401`](https://github.com/apache/spark/commit/f94f401bcfd765b21c3fb466041b42a605d6a814).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85146/testReport)** for PR 20015 at commit [`238d7d4`](https://github.com/apache/spark/commit/238d7d470c583c910bccbca8bbcaa681b67d6025).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    OK. I am fine if you all guys strongly feel about this.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157926202
  
    --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala ---
    @@ -563,6 +563,76 @@ class DateTimeUtilsSuite extends SparkFunSuite {
         }
       }
     
    +  test("truncTimestamp") {
    +    def test(
    --- End diff --
    
    `test` -> `testTrunc` ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Thanks! Merged to master


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85141/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157902164
  
    --- Diff: python/pyspark/sql/functions.py ---
    @@ -1111,6 +1111,24 @@ def trunc(date, format):
         return Column(sc._jvm.functions.trunc(_to_java_column(date), format))
     
     
    +@since(2.3)
    +def date_trunc(format, timestamp):
    +    """
    +    Returns timestamp truncated to the unit specified by the format.
    +
    +    :param format: 'year', 'YYYY', 'yy', 'month', 'mon', 'mm',
    --- End diff --
    
    Also update the original `trunc`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157902098
  
    --- Diff: python/pyspark/sql/functions.py ---
    @@ -1111,6 +1111,24 @@ def trunc(date, format):
         return Column(sc._jvm.functions.trunc(_to_java_column(date), format))
     
     
    +@since(2.3)
    +def date_trunc(format, timestamp):
    +    """
    +    Returns timestamp truncated to the unit specified by the format.
    +
    +    :param format: 'year', 'YYYY', 'yy', 'month', 'mon', 'mm',
    --- End diff --
    
    Nit: `YYYY` -> `yyyy`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20015: [SPARK-22829] Add new built-in function date_trunc()

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20015
  
    **[Test build #85132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85132/testReport)** for PR 20015 at commit [`b12ba92`](https://github.com/apache/spark/commit/b12ba92add942c087dd45933464937479fc24bcd).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait TruncInstant extends BinaryExpression with ImplicitCastInputTypes `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20015: [SPARK-22829] Add new built-in function date_trun...

Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20015#discussion_r157678588
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ---
    @@ -1295,87 +1295,184 @@ case class ParseToTimestamp(left: Expression, format: Option[Expression], child:
       override def dataType: DataType = TimestampType
     }
     
    -/**
    - * Returns date truncated to the unit specified by the format.
    - */
    -// scalastyle:off line.size.limit
    -@ExpressionDescription(
    -  usage = "_FUNC_(date, fmt) - Returns `date` with the time portion of the day truncated to the unit specified by the format model `fmt`.",
    -  examples = """
    -    Examples:
    -      > SELECT _FUNC_('2009-02-12', 'MM');
    -       2009-02-01
    -      > SELECT _FUNC_('2015-10-27', 'YEAR');
    -       2015-01-01
    -  """,
    -  since = "1.5.0")
    -// scalastyle:on line.size.limit
    -case class TruncDate(date: Expression, format: Expression)
    -  extends BinaryExpression with ImplicitCastInputTypes {
    -  override def left: Expression = date
    -  override def right: Expression = format
    -
    -  override def inputTypes: Seq[AbstractDataType] = Seq(DateType, StringType)
    -  override def dataType: DataType = DateType
    +trait TruncTime extends BinaryExpression with ImplicitCastInputTypes {
    +  val time: Expression
    +  val format: Expression
       override def nullable: Boolean = true
    -  override def prettyName: String = "trunc"
     
       private lazy val truncLevel: Int =
         DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
     
    -  override def eval(input: InternalRow): Any = {
    +  /**
    +   *
    +   * @param input
    +   * @param maxLevel Maximum level that can be used for truncation (e.g MONTH for Date input)
    +   * @param truncFunc
    +   * @tparam T
    +   * @return
    +   */
    +  protected def evalHelper[T](input: InternalRow, maxLevel: Int)(
    +    truncFunc: (Any, Int) => T): Any = {
         val level = if (format.foldable) {
           truncLevel
         } else {
           DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
         }
    -    if (level == -1) {
    +    if (level == DateTimeUtils.TRUNC_INVALID || level > maxLevel) {
           // unknown format
           null
         } else {
    -      val d = date.eval(input)
    +      val d = time.eval(input)
           if (d == null) {
             null
           } else {
    -        DateTimeUtils.truncDate(d.asInstanceOf[Int], level)
    +        truncFunc(d, level)
           }
         }
       }
     
    -  override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
    +  protected def codeGenHelper[T](
    --- End diff --
    
    Why do we need a type parameter `T`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org