You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangyum <gi...@git.apache.org> on 2017/05/25 11:01:51 UTC
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/18106
[SPARK-20754][SQL] Support TRUNC (number)
## What changes were proposed in this pull request?
Add support for `TRUNC(number)`, it's similar to Oracle [TRUNC(number)](https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm):
```sql
> SELECT TRUNC(1234567891.1234567891, 4);
1.2345678911234E9
> SELECT TRUNC(1234567891.1234567891, -4);
1.23456E9
> SELECT TRUNC(1234567891.1234567891, 0);
1.234567891E9
> SELECT TRUNC(1234567891.1234567891);
1.234567891E9
```
## How was this patch tested?
unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-20754-trunc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18106.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18106
----
commit a5ade70afe7601db16ec24956f270feb4499ee42
Author: Yuming Wang <wg...@gmail.com>
Date: 2017-05-25T10:21:15Z
Support TRUNC (number)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #83155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83155/testReport)** for PR 18106 at commit [`b59a2df`](https://github.com/apache/spark/commit/b59a2df41d2de5cf658b8dbb01a9fe93d54d768d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum closed the pull request at:
https://github.com/apache/spark/pull/18106
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18106
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18106
https://www.ibm.com/support/knowledgecenter/en/SSEPGG_9.7.0/com.ibm.db2.luw.sql.ref.doc/doc/r0000863.html
I might not have the bandwidth to review this PR until this weekend. @viirya @wzhfy Could you please review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80150/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123831034
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
--- End diff --
Also, I would describe the types without class names (e.g., date).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78094/testReport)** for PR 18106 at commit [`b391b6a`](https://github.com/apache/spark/commit/b391b6a3e51229e501982fe184685d7c2e185172).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78344/testReport)** for PR 18106 at commit [`5456e61`](https://github.com/apache/spark/commit/5456e613a1df28e0b322f24c9aca9470648f96c3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77370/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123828339
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2067,6 +2067,18 @@ object functions {
*/
def radians(columnName: String): Column = radians(Column(columnName))
+ /**
+ * returns number truncated by specified decimal places.
+ *
+ * @param scale: 4. -4, 0
--- End diff --
Let's describe the param.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78507/testReport)** for PR 18106 at commit [`7fee61b`](https://github.com/apache/spark/commit/7fee61b1e084a1ae9966e7ad62b1509085b24151).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78094/testReport)** for PR 18106 at commit [`b391b6a`](https://github.com/apache/spark/commit/b391b6a3e51229e501982fe184685d7c2e185172).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78497 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78497/testReport)** for PR 18106 at commit [`3fd4189`](https://github.com/apache/spark/commit/3fd4189b4c0d000a78f19f3a135f613f9ea0b50a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77394/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r130837526
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,29 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):
--- End diff --
Yes but it brings complexity for both args and kwargs e.g., when both set, method signature in doc and etc. I wonder if it is that important.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18106
R has `trunc` in master/2.3 as well
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78786 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78786/testReport)** for PR 18106 at commit [`f8b1f44`](https://github.com/apache/spark/commit/f8b1f4426836e01e755e73253cb3011d4d6d1bee).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123756528
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
+ 2009-02-01.
+ > SELECT _FUNC_('2015-10-27', 'YEAR');
+ 2015-01-01
+ > SELECT _FUNC_('1989-03-13');
+ 1989-03-01
+ > SELECT _FUNC_(1234567891.1234567891, 4);
+ 1234567891.1234
+ > SELECT _FUNC_(1234567891.1234567891, -4);
+ 1234560000
+ > SELECT _FUNC_(1234567891.1234567891);
+ 1234567891
+ """)
+// scalastyle:on line.size.limit
+case class Trunc(data: Expression, format: Expression)
+ extends BinaryExpression with ExpectsInputTypes {
+
+ def this(data: Expression) = {
+ this(data, Literal(if (data.dataType.isInstanceOf[DateType]) "MM" else 0))
+ }
+
+ override def left: Expression = data
+ override def right: Expression = format
+
+ override def dataType: DataType = data.dataType
+
+ override def inputTypes: Seq[AbstractDataType] = dataType match {
+ case NullType => Seq(dataType, TypeCollection(StringType, IntegerType))
+ case DateType => Seq(dataType, StringType)
+ case DoubleType | DecimalType.Fixed(_, _) => Seq(dataType, IntegerType)
+ case _ => Seq(TypeCollection(DateType, DoubleType, DecimalType),
+ TypeCollection(StringType, IntegerType))
+ }
+
+ override def nullable: Boolean = true
+
+ override def prettyName: String = "trunc"
+
+ private val isTruncNumber =
+ (dataType.isInstanceOf[DoubleType] || dataType.isInstanceOf[DecimalType]) &&
+ format.dataType.isInstanceOf[IntegerType]
+ private val isTruncDate =
+ dataType.isInstanceOf[DateType] && format.dataType.isInstanceOf[StringType]
+
+ private lazy val truncFormat: Int = if (isTruncNumber) {
+ format.eval().asInstanceOf[Int]
+ } else if (isTruncDate) {
+ DateTimeUtils.parseTruncLevel(format.eval().asInstanceOf[UTF8String])
+ } else {
+ 0
+ }
+
+ override def eval(input: InternalRow): Any = {
--- End diff --
override `nullSafeEval`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77399/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123757175
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
+ 2009-02-01.
+ > SELECT _FUNC_('2015-10-27', 'YEAR');
+ 2015-01-01
+ > SELECT _FUNC_('1989-03-13');
+ 1989-03-01
+ > SELECT _FUNC_(1234567891.1234567891, 4);
+ 1234567891.1234
+ > SELECT _FUNC_(1234567891.1234567891, -4);
+ 1234560000
+ > SELECT _FUNC_(1234567891.1234567891);
+ 1234567891
+ """)
+// scalastyle:on line.size.limit
+case class Trunc(data: Expression, format: Expression)
+ extends BinaryExpression with ExpectsInputTypes {
+
+ def this(data: Expression) = {
+ this(data, Literal(if (data.dataType.isInstanceOf[DateType]) "MM" else 0))
+ }
+
+ override def left: Expression = data
+ override def right: Expression = format
+
+ override def dataType: DataType = data.dataType
+
+ override def inputTypes: Seq[AbstractDataType] = dataType match {
+ case NullType => Seq(dataType, TypeCollection(StringType, IntegerType))
+ case DateType => Seq(dataType, StringType)
+ case DoubleType | DecimalType.Fixed(_, _) => Seq(dataType, IntegerType)
+ case _ => Seq(TypeCollection(DateType, DoubleType, DecimalType),
+ TypeCollection(StringType, IntegerType))
+ }
+
+ override def nullable: Boolean = true
+
+ override def prettyName: String = "trunc"
+
+ private val isTruncNumber =
+ (dataType.isInstanceOf[DoubleType] || dataType.isInstanceOf[DecimalType]) &&
+ format.dataType.isInstanceOf[IntegerType]
--- End diff --
Combined with this and `inputTypes`, once we have input types like (DoubleType, StringType), looks like `eval` simply return null. Seems to me it's not considered as correct result.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123798230
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -1382,8 +1382,8 @@ test_that("column functions", {
c20 <- to_timestamp(c) + to_timestamp(c, "yyyy") + to_date(c, "yyyy")
c21 <- posexplode_outer(c) + explode_outer(c)
c22 <- not(c)
- c23 <- trunc(c, "year") + trunc(c, "yyyy") + trunc(c, "yy") +
- trunc(c, "month") + trunc(c, "mon") + trunc(c, "mm")
+ c23 <- trunc(to_date(c), "year") + trunc(to_date(c), "yyyy") + trunc(to_date(c), "yy") +
+ trunc(to_date(c), "month") + trunc(to_date(c), "mon") + trunc(to_date(c), "mm")
--- End diff --
R should support both date and number too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78786/testReport)** for PR 18106 at commit [`f8b1f44`](https://github.com/apache/spark/commit/f8b1f4426836e01e755e73253cb3011d4d6d1bee).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123920115
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
--- End diff --
Yes, This is what I worry about.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123822928
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -1382,8 +1382,8 @@ test_that("column functions", {
c20 <- to_timestamp(c) + to_timestamp(c, "yyyy") + to_date(c, "yyyy")
c21 <- posexplode_outer(c) + explode_outer(c)
c22 <- not(c)
- c23 <- trunc(c, "year") + trunc(c, "yyyy") + trunc(c, "yy") +
- trunc(c, "month") + trunc(c, "mon") + trunc(c, "mm")
+ c23 <- trunc(to_date(c), "year") + trunc(to_date(c), "yyyy") + trunc(to_date(c), "yy") +
+ trunc(to_date(c), "month") + trunc(to_date(c), "mon") + trunc(to_date(c), "mm")
--- End diff --
Ah, just to be clear, I meant https://github.com/apache/spark/pull/18106/files/7fee61b1e084a1ae9966e7ad62b1509085b24151#r123758795The current state looks drop supporting this function with string that can be implicitly casted into date and `to_date` is required which breaks backward compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #80066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80066/testReport)** for PR 18106 at commit [`f8b1f44`](https://github.com/apache/spark/commit/f8b1f4426836e01e755e73253cb3011d4d6d1bee).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78094/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78493/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #80152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80152/testReport)** for PR 18106 at commit [`3d40c36`](https://github.com/apache/spark/commit/3d40c366892303cd0de8259b31aebe7a748d89e6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78273/testReport)** for PR 18106 at commit [`5456e61`](https://github.com/apache/spark/commit/5456e613a1df28e0b322f24c9aca9470648f96c3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wzhfy <gi...@git.apache.org>.
Github user wzhfy commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123871820
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala ---
@@ -44,4 +46,49 @@ class MiscExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
assert(evaluate(Uuid()) !== evaluate(Uuid()))
}
+ test("trunc") {
+ // numeric
+ def testTruncNumber(input: Double, fmt: Int, expected: Double): Unit = {
+ checkEvaluation(Trunc(Literal.create(input, DoubleType),
+ Literal.create(fmt, IntegerType)),
+ expected)
+ checkEvaluation(Trunc(Literal.create(input, DoubleType),
+ NonFoldableLiteral.create(fmt, IntegerType)),
+ expected)
+ }
+
+ testTruncNumber(1234567891.1234567891, 4, 1234567891.1234)
+ testTruncNumber(1234567891.1234567891, -4, 1234560000)
+ testTruncNumber(1234567891.1234567891, 0, 1234567891)
+
+ checkEvaluation(Trunc(Literal.create(1D, DoubleType),
+ NonFoldableLiteral.create(null, IntegerType)),
+ null)
+ checkEvaluation(Trunc(Literal.create(null, DoubleType),
+ NonFoldableLiteral.create(1, IntegerType)),
+ null)
+ checkEvaluation(Trunc(Literal.create(null, DoubleType),
+ NonFoldableLiteral.create(null, IntegerType)),
+ null)
+
+ // date
--- End diff --
Shall we split this test into two tests for numeric and date respectively?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r130833154
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,29 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):
--- End diff --
We can work around this with kwargs if it's important to change the name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #77361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77361/testReport)** for PR 18106 at commit [`a5ade70`](https://github.com/apache/spark/commit/a5ade70afe7601db16ec24956f270feb4499ee42).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78693/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123755745
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
+ 2009-02-01.
+ > SELECT _FUNC_('2015-10-27', 'YEAR');
+ 2015-01-01
+ > SELECT _FUNC_('1989-03-13');
+ 1989-03-01
+ > SELECT _FUNC_(1234567891.1234567891, 4);
+ 1234567891.1234
+ > SELECT _FUNC_(1234567891.1234567891, -4);
+ 1234560000
+ > SELECT _FUNC_(1234567891.1234567891);
+ 1234567891
+ """)
+// scalastyle:on line.size.limit
+case class Trunc(data: Expression, format: Expression)
+ extends BinaryExpression with ExpectsInputTypes {
+
+ def this(data: Expression) = {
+ this(data, Literal(if (data.dataType.isInstanceOf[DateType]) "MM" else 0))
+ }
+
+ override def left: Expression = data
+ override def right: Expression = format
+
+ override def dataType: DataType = data.dataType
+
+ override def inputTypes: Seq[AbstractDataType] = dataType match {
+ case NullType => Seq(dataType, TypeCollection(StringType, IntegerType))
+ case DateType => Seq(dataType, StringType)
+ case DoubleType | DecimalType.Fixed(_, _) => Seq(dataType, IntegerType)
+ case _ => Seq(TypeCollection(DateType, DoubleType, DecimalType),
--- End diff --
Do we need to have this case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78493/testReport)** for PR 18106 at commit [`88d1e38`](https://github.com/apache/spark/commit/88d1e38c8fe55c52aaa64754b66c56472141a682).
* This patch **fails SparkR unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123752789
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
+ 2009-02-01.
+ > SELECT _FUNC_('2015-10-27', 'YEAR');
+ 2015-01-01
+ > SELECT _FUNC_('1989-03-13');
+ 1989-03-01
+ > SELECT _FUNC_(1234567891.1234567891, 4);
+ 1234567891.1234
+ > SELECT _FUNC_(1234567891.1234567891, -4);
+ 1234560000
+ > SELECT _FUNC_(1234567891.1234567891);
+ 1234567891
+ """)
+// scalastyle:on line.size.limit
+case class Trunc(data: Expression, format: Expression)
--- End diff --
truncParam?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78497/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78325/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18106
Although then we can't use just one `trunc` function, it seems ok for me because not all databases use `trunc` to truncate both number and datetime.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #80159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80159/testReport)** for PR 18106 at commit [`931f07d`](https://github.com/apache/spark/commit/931f07de787081cdd6822dbf396ec1b8d205f25e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123161910
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,141 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
+ 2009-02-01.
+ > SELECT _FUNC_('2015-10-27', 'YEAR');
+ 2015-01-01
+ > SELECT _FUNC_('1989-03-13');
+ 1989-03-01
+ > SELECT _FUNC_(1234567891.1234567891, 4);
+ 1234567891.1234
+ > SELECT _FUNC_(1234567891.1234567891, -4);
+ 1234560000
+ > SELECT _FUNC_(1234567891.1234567891);
+ 1234567891
+ """)
+// scalastyle:on line.size.limit
+case class Trunc(data: Expression, format: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ def this(data: Expression) = {
+ this(data, Literal(if (data.dataType.isInstanceOf[DateType]) "MM" else 0))
+ }
+
+ override def left: Expression = data
+ override def right: Expression = format
+
+ val isTruncNumber = format.dataType.isInstanceOf[IntegerType]
+
+ override def dataType: DataType = data.dataType
+
+ override def inputTypes: Seq[AbstractDataType] =
+ Seq(TypeCollection(DateType, DoubleType, DecimalType),
+ TypeCollection(StringType, IntegerType))
--- End diff --
If we are going to have only `trunc` for truncating number and datetime. We should prevent wrong input types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #77361 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77361/testReport)** for PR 18106 at commit [`a5ade70`](https://github.com/apache/spark/commit/a5ade70afe7601db16ec24956f270feb4499ee42).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Trunc(data: Expression, format: Expression = Literal(0))`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123828403
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2067,6 +2067,18 @@ object functions {
*/
def radians(columnName: String): Column = radians(Column(columnName))
+ /**
+ * returns number truncated by specified decimal places.
--- End diff --
little nit: `returns` -> `Returns`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123884847
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,28 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, format):
"""
- Returns date truncated to the unit specified by the format.
+ Returns date truncated to the unit specified by the format or
+ number truncated by specified decimal places.
:param format: 'year', 'YYYY', 'yy' or 'month', 'mon', 'mm'
>>> df = spark.createDataFrame([('1997-02-28',)], ['d'])
- >>> df.select(trunc(df.d, 'year').alias('year')).collect()
+ >>> df.select(trunc(to_date(df.d), 'year').alias('year')).collect()
[Row(year=datetime.date(1997, 1, 1))]
- >>> df.select(trunc(df.d, 'mon').alias('month')).collect()
+ >>> df.select(trunc(to_date(df.d), 'mon').alias('month')).collect()
--- End diff --
this, could be a bigger problem :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123758531
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
--- End diff --
As it doesn't extends `ImplicitCastInputTypes` anymore, I think you can't directly use string as date parameter?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83158/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123752317
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,28 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, format):
"""
- Returns date truncated to the unit specified by the format.
+ Returns date truncated to the unit specified by the format or
+ number truncated by specified decimal places.
:param format: 'year', 'YYYY', 'yy' or 'month', 'mon', 'mm'
--- End diff --
And we need to update this param doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/18106
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78493/testReport)** for PR 18106 at commit [`88d1e38`](https://github.com/apache/spark/commit/88d1e38c8fe55c52aaa64754b66c56472141a682).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78362/testReport)** for PR 18106 at commit [`5456e61`](https://github.com/apache/spark/commit/5456e613a1df28e0b322f24c9aca9470648f96c3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123827347
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2067,6 +2067,18 @@ object functions {
*/
def radians(columnName: String): Column = radians(Column(columnName))
+ /**
+ * returns number truncated by specified decimal places.
+ *
+ * @param scale: 4. -4, 0
+ *
+ * @group math_funcs
+ * @since 2.3.0
+ */
+ def trunc(db: Column, scale: Int = 0): Column = withExpr {
--- End diff --
I would avoid using default value here unless we are very sure on this (e.g., other languages can call this without any problem, this guarantees method signature and does not break binary compatibility and etc.).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #77394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77394/testReport)** for PR 18106 at commit [`e7e6e5b`](https://github.com/apache/spark/commit/e7e6e5b436be61b174253ea6c520a8fe2ad87383).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18106
@dongjoon-hyun Actually `TRUNC (number)` not resolved. I will fix it soon.
https://issues.apache.org/jira/browse/SPARK-23906
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #80150 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80150/testReport)** for PR 18106 at commit [`3d40c36`](https://github.com/apache/spark/commit/3d40c366892303cd0de8259b31aebe7a748d89e6).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/18106
@wangyum .
https://issues.apache.org/jira/browse/SPARK-20754 is resolved by you. Could you close this PR?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78693/testReport)** for PR 18106 at commit [`f8b1f44`](https://github.com/apache/spark/commit/f8b1f4426836e01e755e73253cb3011d4d6d1bee).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123190922
--- Diff: sql/core/src/test/resources/sql-tests/inputs/operators.sql ---
@@ -92,3 +91,7 @@ select abs(-3.13), abs('-2.19');
-- positive/negative
select positive('-1.11'), positive(-1.11), negative('-1.11'), negative(-1.11);
+
+-- trunc number
+select trunc(1234567891.1234567891, 4), trunc(1234567891.1234567891, -4), trunc(1234567891.1234567891, 0), trunc(1234567891.1234567891);
+select trunc(1234567891.1234567891, null), trunc(null, 4), trunc(null, null);
--- End diff --
Can you add test cases like `trunc(1234567891.1234567891, "yyyy")` and `trunc('2015-07-22', 4)`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123754453
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
--- End diff --
Please also describe default values (MM or 0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #80159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80159/testReport)** for PR 18106 at commit [`931f07d`](https://github.com/apache/spark/commit/931f07de787081cdd6822dbf396ec1b8d205f25e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18106
@actuaryzhang
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78453/testReport)** for PR 18106 at commit [`d40a46f`](https://github.com/apache/spark/commit/d40a46fbbd3c69f590ca5f0d604da84d218fef2e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/18106
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18106
Retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78057/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18106
cc @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78453/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18106
@gatorsmile I'll review this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18106
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78362/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78693/testReport)** for PR 18106 at commit [`f8b1f44`](https://github.com/apache/spark/commit/f8b1f4426836e01e755e73253cb3011d4d6d1bee).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123155483
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,141 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
+ 2009-02-01.
+ > SELECT _FUNC_('2015-10-27', 'YEAR');
+ 2015-01-01
+ > SELECT _FUNC_('1989-03-13');
+ 1989-03-01
+ > SELECT _FUNC_(1234567891.1234567891, 4);
+ 1234567891.1234
+ > SELECT _FUNC_(1234567891.1234567891, -4);
+ 1234560000
+ > SELECT _FUNC_(1234567891.1234567891);
+ 1234567891
+ """)
+// scalastyle:on line.size.limit
+case class Trunc(data: Expression, format: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+
+ def this(data: Expression) = {
+ this(data, Literal(if (data.dataType.isInstanceOf[DateType]) "MM" else 0))
+ }
+
+ override def left: Expression = data
+ override def right: Expression = format
+
+ val isTruncNumber = format.dataType.isInstanceOf[IntegerType]
+
+ override def dataType: DataType = data.dataType
+
+ override def inputTypes: Seq[AbstractDataType] =
+ Seq(TypeCollection(DateType, DoubleType, DecimalType),
+ TypeCollection(StringType, IntegerType))
--- End diff --
I think this might lead to wrong input types combinations such as (DoubleType, StringType) and (DateType, IntegerType)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #83158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83158/testReport)** for PR 18106 at commit [`679ff98`](https://github.com/apache/spark/commit/679ff9838cbf7c5e3d5acb19f4a1a6a4bde75d6a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ClearCacheCommand() extends RunnableCommand `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77361/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123831200
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
--- End diff --
I guess this also drops the support of other types (e.g., timestamp) basically as we don't allow implicit cast (e.g, `SELECT trunc(timestamp('2009-02-12'), 'MM')`)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123824736
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,28 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, format):
"""
- Returns date truncated to the unit specified by the format.
+ Returns date truncated to the unit specified by the format or
+ number truncated by specified decimal places.
:param format: 'year', 'YYYY', 'yy' or 'month', 'mon', 'mm'
>>> df = spark.createDataFrame([('1997-02-28',)], ['d'])
- >>> df.select(trunc(df.d, 'year').alias('year')).collect()
+ >>> df.select(trunc(to_date(df.d), 'year').alias('year')).collect()
[Row(year=datetime.date(1997, 1, 1))]
- >>> df.select(trunc(df.d, 'mon').alias('month')).collect()
+ >>> df.select(trunc(to_date(df.d), 'mon').alias('month')).collect()
--- End diff --
Here too. If this change is needed to pass Pyhon tests, it looks breaking the compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #80066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80066/testReport)** for PR 18106 at commit [`f8b1f44`](https://github.com/apache/spark/commit/f8b1f4426836e01e755e73253cb3011d4d6d1bee).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #77399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77399/testReport)** for PR 18106 at commit [`7157820`](https://github.com/apache/spark/commit/7157820ae48528059e8008535a37ae18b6df5530).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wzhfy <gi...@git.apache.org>.
Github user wzhfy commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123871837
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala ---
@@ -44,4 +46,49 @@ class MiscExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
assert(evaluate(Uuid()) !== evaluate(Uuid()))
}
+ test("trunc") {
+ // numeric
+ def testTruncNumber(input: Double, fmt: Int, expected: Double): Unit = {
+ checkEvaluation(Trunc(Literal.create(input, DoubleType),
+ Literal.create(fmt, IntegerType)),
+ expected)
+ checkEvaluation(Trunc(Literal.create(input, DoubleType),
+ NonFoldableLiteral.create(fmt, IntegerType)),
+ expected)
+ }
+
+ testTruncNumber(1234567891.1234567891, 4, 1234567891.1234)
+ testTruncNumber(1234567891.1234567891, -4, 1234560000)
+ testTruncNumber(1234567891.1234567891, 0, 1234567891)
--- End diff --
Also check testTruncNumber(0.1234567891, -1, 0.1234567891)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123758795
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
--- End diff --
And I don't think we should drop this support.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78080/testReport)** for PR 18106 at commit [`b391b6a`](https://github.com/apache/spark/commit/b391b6a3e51229e501982fe184685d7c2e185172).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123684235
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,28 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, format):
"""
- Returns date truncated to the unit specified by the format.
+ Returns date truncated to the unit specified by the format or
+ number truncated by specified decimal places.
--- End diff --
in the latter case where `data` is a number, the 2nd parameter as called `format` seems a bit out of place?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78057/testReport)** for PR 18106 at commit [`3d92a48`](https://github.com/apache/spark/commit/3d92a48a54c5bf0222c032fc5c205ea1792630a2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123753748
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
--- End diff --
fmt -> expr?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78786/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80159/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #83155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83155/testReport)** for PR 18106 at commit [`b59a2df`](https://github.com/apache/spark/commit/b59a2df41d2de5cf658b8dbb01a9fe93d54d768d).
* This patch **fails PySpark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r124328743
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,29 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):
--- End diff --
@ueshin @holdenk re: changing param name in python.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #77370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77370/testReport)** for PR 18106 at commit [`c63856b`](https://github.com/apache/spark/commit/c63856b666e0992bf464c726f78b02e2521a41b5).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18106
Is there duplicated codes between trunc(number) and trunc(date)? If no, seems to me we don't necessarily let one expression to have two different features.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123759792
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -1382,8 +1382,8 @@ test_that("column functions", {
c20 <- to_timestamp(c) + to_timestamp(c, "yyyy") + to_date(c, "yyyy")
c21 <- posexplode_outer(c) + explode_outer(c)
c22 <- not(c)
- c23 <- trunc(c, "year") + trunc(c, "yyyy") + trunc(c, "yy") +
- trunc(c, "month") + trunc(c, "mon") + trunc(c, "mm")
+ c23 <- trunc(to_date(c), "year") + trunc(to_date(c), "yyyy") + trunc(to_date(c), "yy") +
+ trunc(to_date(c), "month") + trunc(to_date(c), "mon") + trunc(to_date(c), "mm")
--- End diff --
y. I don't think we should do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78273/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #77370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77370/testReport)** for PR 18106 at commit [`c63856b`](https://github.com/apache/spark/commit/c63856b666e0992bf464c726f78b02e2521a41b5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18106
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r130801952
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,29 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):
--- End diff --
@wangyum, would you mind revert this renaming? This breaks the compatibility if user script calls this by
```python
trunc(..., format= ...)
trunc(date=..., format= ...)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18106
I'll fix it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83155/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #77399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77399/testReport)** for PR 18106 at commit [`7157820`](https://github.com/apache/spark/commit/7157820ae48528059e8008535a37ae18b6df5530).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18106
Yes, I also tend to be one `trunc` function. I found Presto provides two functions `date_trunc` and `truncate`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78325/testReport)** for PR 18106 at commit [`5456e61`](https://github.com/apache/spark/commit/5456e613a1df28e0b322f24c9aca9470648f96c3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #80150 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80150/testReport)** for PR 18106 at commit [`3d40c36`](https://github.com/apache/spark/commit/3d40c366892303cd0de8259b31aebe7a748d89e6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123884830
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -1382,8 +1382,8 @@ test_that("column functions", {
c20 <- to_timestamp(c) + to_timestamp(c, "yyyy") + to_date(c, "yyyy")
c21 <- posexplode_outer(c) + explode_outer(c)
c22 <- not(c)
- c23 <- trunc(c, "year") + trunc(c, "yyyy") + trunc(c, "yy") +
- trunc(c, "month") + trunc(c, "mon") + trunc(c, "mm")
+ c23 <- trunc(to_date(c), "year") + trunc(to_date(c), "yyyy") + trunc(to_date(c), "yy") +
+ trunc(to_date(c), "month") + trunc(to_date(c), "mon") + trunc(to_date(c), "mm")
--- End diff --
that's a good point. fortunately (?) trunc was only added to R in 2.3.0, so I think we need to make sure (manually, add unit test) that trunc works on date columns and numeric columns
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123919660
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala ---
@@ -132,3 +133,154 @@ case class Uuid() extends LeafExpression {
s"UTF8String.fromString(java.util.UUID.randomUUID().toString());", isNull = "false")
}
}
+
+/**
+ * Returns date truncated to the unit specified by the format or
+ * numeric truncated to scale decimal places.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = """
+ _FUNC_(data[, fmt]) - Returns `data` truncated by the format model `fmt`.
+ If `data` is DateType, returns `data` with the time portion of the day truncated to the unit specified by the format model `fmt`.
+ If `data` is DecimalType/DoubleType, returns `data` truncated to `fmt` decimal places.
+ """,
+ extended = """
+ Examples:
+ > SELECT _FUNC_('2009-02-12', 'MM');
+ 2009-02-01.
+ > SELECT _FUNC_('2015-10-27', 'YEAR');
+ 2015-01-01
+ > SELECT _FUNC_('1989-03-13');
+ 1989-03-01
+ > SELECT _FUNC_(1234567891.1234567891, 4);
+ 1234567891.1234
+ > SELECT _FUNC_(1234567891.1234567891, -4);
+ 1234560000
+ > SELECT _FUNC_(1234567891.1234567891);
+ 1234567891
+ """)
+// scalastyle:on line.size.limit
+case class Trunc(data: Expression, format: Expression)
+ extends BinaryExpression with ExpectsInputTypes {
+
+ def this(data: Expression) = {
+ this(data, Literal(if (data.dataType.isInstanceOf[DateType]) "MM" else 0))
+ }
+
+ override def left: Expression = data
+ override def right: Expression = format
+
+ override def dataType: DataType = data.dataType
+
+ override def inputTypes: Seq[AbstractDataType] = dataType match {
+ case NullType => Seq(dataType, TypeCollection(StringType, IntegerType))
+ case DateType => Seq(dataType, StringType)
+ case DoubleType | DecimalType.Fixed(_, _) => Seq(dataType, IntegerType)
+ case _ => Seq(TypeCollection(DateType, DoubleType, DecimalType),
--- End diff --
Add this case to show all supported type:
```
> select trunc(false, 'MON');
Error in query: cannot resolve 'trunc(false, 'MON')' due to data type mismatch: argument 1 requires (date or double or decimal) type, however, 'false' is of boolean type.; line 1 pos 7;
'Project [unresolvedalias(trunc(false, MON), None)]
+- OneRowRelation$
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78362/testReport)** for PR 18106 at commit [`5456e61`](https://github.com/apache/spark/commit/5456e613a1df28e0b322f24c9aca9470648f96c3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #77394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77394/testReport)** for PR 18106 at commit [`e7e6e5b`](https://github.com/apache/spark/commit/e7e6e5b436be61b174253ea6c520a8fe2ad87383).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/18106
I'll add @gatorsmile since this is SQL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18106
This is close to merge. Could you resolve the conflicts? Then, I will review it. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r124329456
--- Diff: python/pyspark/sql/functions.py ---
@@ -1028,20 +1028,29 @@ def to_timestamp(col, format=None):
@since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):
--- End diff --
I believe this definitely breaks backward compatibility for keyword-argument usage in Python.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #83158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83158/testReport)** for PR 18106 at commit [`679ff98`](https://github.com/apache/spark/commit/679ff9838cbf7c5e3d5acb19f4a1a6a4bde75d6a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18106
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/18106
+100, @wangyum . Thanks. :)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80066/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80152/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18106
ping @wangyum
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #80152 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80152/testReport)** for PR 18106 at commit [`3d40c36`](https://github.com/apache/spark/commit/3d40c366892303cd0de8259b31aebe7a748d89e6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78057 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78057/testReport)** for PR 18106 at commit [`3d92a48`](https://github.com/apache/spark/commit/3d92a48a54c5bf0222c032fc5c205ea1792630a2).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Trunc(data: Expression, format: Expression)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78080/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18106#discussion_r123759339
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -1382,8 +1382,8 @@ test_that("column functions", {
c20 <- to_timestamp(c) + to_timestamp(c, "yyyy") + to_date(c, "yyyy")
c21 <- posexplode_outer(c) + explode_outer(c)
c22 <- not(c)
- c23 <- trunc(c, "year") + trunc(c, "yyyy") + trunc(c, "yy") +
- trunc(c, "month") + trunc(c, "mon") + trunc(c, "mm")
+ c23 <- trunc(to_date(c), "year") + trunc(to_date(c), "yyyy") + trunc(to_date(c), "yy") +
+ trunc(to_date(c), "month") + trunc(to_date(c), "mon") + trunc(to_date(c), "mm")
--- End diff --
Isn't this actually behaviour change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78453/testReport)** for PR 18106 at commit [`d40a46f`](https://github.com/apache/spark/commit/d40a46fbbd3c69f590ca5f0d604da84d218fef2e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18106
**[Test build #78497 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78497/testReport)** for PR 18106 at commit [`3fd4189`](https://github.com/apache/spark/commit/3fd4189b4c0d000a78f19f3a135f613f9ea0b50a).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18106: [SPARK-20754][SQL] Support TRUNC (number)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18106
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78507/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org