You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gczsjdy <gi...@git.apache.org> on 2017/01/05 10:50:37 UTC
[GitHub] spark pull request #16476: Implement expression field
GitHub user gczsjdy opened a pull request:
https://github.com/apache/spark/pull/16476
Implement expression field
This is an implementation of Hive's `field`.
field(str, str1, str2, ... ) is a variable-length(>=2) function which returns the index of str in the list (str1, str2, ... ) or 0 if not found.
Every parameter is required to be subtype of AtomicType.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gczsjdy/spark udffield
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16476.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16476
----
commit 5f650641858653b42e08a73c87daf8af0b7cdd2b
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-21T12:30:17Z
Add field expression
commit b9d589f78c287c5155f668b726c257465a1e0327
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-21T12:33:13Z
Add field expression unit test
commit 0bb83282b36cee3e6edc15e991a65ffb7de54474
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-28T13:00:40Z
Add doGenCode and modify the function description
commit 255b177b7c1a81e3f79b175498c754e2c549791e
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-28T13:25:24Z
Add field expression unit test cases
commit bb5a3ef79ada230660ae867f07af7c8d3a58d43b
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-28T13:31:18Z
Modify function description
commit f291815544fac747cb26482ec2144d4f1bc91680
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-29T06:23:36Z
Add support for field children which is not List
commit f07ec52d745bc379153a56333638f957748139d1
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-29T06:29:30Z
Add field function API for DataFrame and unit test cases
commit 6319dc5e091a6b593e22a172397421955811c6b6
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-29T07:03:40Z
Modify @since version
commit df1038d6665c1c084a0ef0a1ac9efa0ec8f5d52d
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-29T07:04:19Z
Add Python API
commit 38fe71d90b85b2e8a3e0badda0a40ae411713b69
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-29T08:31:43Z
Add Apache License
commit 3fc66d5af7030baebbe14f0b21f2afd2ff9f231b
Author: GuoChenzhao <ch...@intel.com>
Date: 2016-12-30T05:47:26Z
Modify import order
commit 5c55f42db084e5777704583d36d018da288e76f2
Author: GuoChenzhao <ch...@intel.com>
Date: 2017-01-05T05:48:33Z
Move Field class def to ConditionalExpressions and modify the test
commit ce177f0f0c1ae9c21d72f2f9a4180718a74271e4
Author: GuoChenzhao <ch...@intel.com>
Date: 2017-01-05T05:58:40Z
Modify the line feed's location
commit 57229a99baa2da09167a62291841eafada78f3e0
Author: GuoChenzhao <ch...@intel.com>
Date: 2017-01-05T06:17:57Z
Move Field function test to ColumnExpressionSuite
commit 71e54da58f7ccaad187bf5c693111c2ce076faf2
Author: GuoChenzhao <ch...@intel.com>
Date: 2017-01-05T08:56:21Z
Add null support and corresponding test cases
commit 83687c8c3bc38dc221dc42595ba83e992d0cf36e
Author: GuoChenzhao <ch...@intel.com>
Date: 2017-01-05T09:09:40Z
Remove blank lines
commit 192fadc68a63552dad40a1fe40e664425cd55146
Author: GuoChenzhao <ch...@intel.com>
Date: 2017-01-05T10:18:46Z
Modify field function definition to require >= 2 params
commit 08ab497582344158df7b0fbb95808f5444e1aa81
Author: GuoChenzhao <ch...@intel.com>
Date: 2017-01-05T10:22:48Z
Add blank line
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #75038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75038/testReport)** for PR 16476 at commit [`9b4155a`](https://github.com/apache/spark/commit/9b4155a42796d4aa3224d109f28822196bf63041).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r99535960
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,99 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.length) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value)) {
+ dataTypeMatchIndex(index)
+ } else {
+ findEqual(index + 1)
+ }
+ }
+ }
+ if (target == null) 0 else findEqual(index = 0)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val dataTypes = children.map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, _), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
+ s"""
+ ${code1}
+ else {
+ ${code2}
+ }
+ """
+ }
+
+ ev.copy(code = s"""
+ ${target.code}
--- End diff --
Thanks, I have added a if to bypass the case when target.isNull ==true, which is actually the same as yours.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74639/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101516174
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,104 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else {
+ TypeCheckResult.TypeCheckSuccess
+ }
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.length) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value)) {
+ dataTypeMatchIndex(index)
+ } else {
+ findEqual(index + 1)
+ }
+ }
+ }
+ if (target == null) 0 else findEqual(index = 0)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
--- End diff --
I think you already call `genCode` here, and only filter type mismatch `ExprCode` in line 441.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72553/testReport)** for PR 16476 at commit [`d22dbb3`](https://github.com/apache/spark/commit/d22dbb345d7d48bae9a69e09d4049be58df6af61).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95055000
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
--- End diff --
- nit: space after `if`
- checkstyle will fail saying if-else needs to use braces
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #74009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74009/testReport)** for PR 16476 at commit [`dd598a4`](https://github.com/apache/spark/commit/dd598a40429493e5846009b33b0aa45043ee688a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #74655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74655/testReport)** for PR 16476 at commit [`fc47e0f`](https://github.com/apache/spark/commit/fc47e0fbb8f196c91e69493d44a30a5b09a68f37).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/16476
@gczsjdy Can you resolve the conflict?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #93594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93594/testReport)** for PR 16476 at commit [`c75d786`](https://github.com/apache/spark/commit/c75d786aef5632098a78be3640a67463374e9c8b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101438354
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
Nit: Maybe we can use `foldLeft` to replace current approach to get rid of the floating `else`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72554/testReport)** for PR 16476 at commit [`c23deb5`](https://github.com/apache/spark/commit/c23deb579d3ff0d3bb432484c2a8cce487bb2fa2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95060564
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
--- End diff --
@tejasapatil The first 3 strings are base test strings, the 5th is for null of string type. Probably I can remove the 4th one which is not useful.
About the tests' role, could you please check another thread where I @ you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #93100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93100/testReport)** for PR 16476 at commit [`4b63e94`](https://github.com/apache/spark/commit/4b63e945739ac0c9d2dc572354edc97e2025683e).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93594/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@cloud-fan I have checked some other popular RDBMS, such as Oracle, Microsoft SQL Server, PostgreSQL, DB2, SQLite, neither of them support function `FIELD`.
In my opinion, Hive has an important place in SQL on Big Data area, and many Spark SQL users originally use Hive, so maybe it's more reasonable to follow Hive since other databases currently not support this function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95732892
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
--- End diff --
So basically why we only support AtomicType and NullType is mainly due to being consistent with Hive?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16476
@gczsjdy I am not sure where you find the definition of Hive's `field` function. But in this link: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions
Looks like it behaves very similar to MySQL. It doesn't claim that the arguments of different types won't be compared.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95055128
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
From the name it felt like this method would put `code1` in `if` block and `code2` in `else` block but turns out thats not the case. That floating `else` looks weird.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101673768
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
For now, I use `reduceRight`, which I think is a 'special case' function of `foldRight`.
If I understand your meaning of floating `else` right(could you please explain it a little bit?), these 2 functions both can't avoid floating `else`, because we need nested `else` in `else` block, like this:
`if (xxx)
else {
if (xxx)
else {
...
}
}
`, so if we avoid floating `else` in `genIfElseStructure`, `else` should be in `updateEval`, which will make the code unclear and complicated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72994 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72994/testReport)** for PR 16476 at commit [`0665a1d`](https://github.com/apache/spark/commit/0665a1d0d76bdb553010775eb1c53b8d461e792a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054994
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
--- End diff --
Any reason to not do this iteratively ? I would suggest avoiding recursion to get better perf.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72505/testReport)** for PR 16476 at commit [`66e779e`](https://github.com/apache/spark/commit/66e779efd8aeb0027427577fbe225893e47bd419).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95024068
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
--- End diff --
can we use strings as examples rather than integer literals?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #74639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74639/testReport)** for PR 16476 at commit [`4e60b7c`](https://github.com/apache/spark/commit/4e60b7c52c0ca9e20296256607ce78741d80cea3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait ImplicitCastInputTypesToSameType extends ExpectsInputTypes `
* `case class Field(children: Seq[Expression]) extends Expression`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95102912
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
--- End diff --
@chenghao-intel I have added that annotation and remove the `toList`.
Still working on the rule for folding not match datatypes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72431/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95292505
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
+ _._1.dataType == children.head.dataType).map(_._2)
--- End diff --
Got it, thx.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@HyukjinKwon Done, thanks : )
Ping @maropu
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95718514
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
--- End diff --
search string -> search expression?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95060436
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
+ val str1 = Literal("\u82b1\u82b1\u4e16\u754c")
+ val str2 = Literal("a")
+ val str3 = Literal("b")
+ val str4 = Literal("")
+ val str5 = Literal("999")
+ val strNull = Literal.create(null, StringType)
+
+ val bool1 = Literal(true)
+ val bool2 = Literal(false)
+
+ val int1 = Literal(1)
+ val int2 = Literal(2)
+ val int3 = Literal(3)
+ val int4 = Literal(999)
+ val intNull = Literal.create(null, IntegerType)
+
+ val double1 = Literal(1.221)
+ val double2 = Literal(1.222)
+ val double3 = Literal(1.224)
+
+ val timeStamp1 = Literal(new Timestamp(2016, 12, 27, 14, 22, 1, 1))
+ val timeStamp2 = Literal(new Timestamp(1988, 6, 3, 1, 1, 1, 1))
+ val timeStamp3 = Literal(new Timestamp(1990, 6, 5, 1, 1, 1, 1))
+
+ val date1 = Literal(new Date(1949, 1, 1))
+ val date2 = Literal(new Date(1979, 1, 1))
+ val date3 = Literal(new Date(1989, 1, 1))
+
+ checkEvaluation(Field(Seq(str1, str2, str3, str1)), 3)
+ checkEvaluation(Field(Seq(str2, str2, str2, str1)), 1)
+ checkEvaluation(Field(Seq(str4, str4, str4, str1)), 1)
+ checkEvaluation(Field(Seq(bool1, bool2, bool1, bool1)), 2)
+ checkEvaluation(Field(Seq(int1, int2, int3, int1)), 3)
+ checkEvaluation(Field(Seq(double2, double3, double1, double2)), 3)
+ checkEvaluation(Field(Seq(timeStamp1, timeStamp2, timeStamp3, timeStamp1)), 3)
+ checkEvaluation(Field(Seq(date1, date1, date2, date3)), 1)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int4)), 6)
+ checkEvaluation(Field(Seq(str5, str1, str2, str4)), 0)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(int1, strNull, intNull, bool1, date1, timeStamp2, int3)), 0)
--- End diff --
Line 180 is to test multi types of parameters;
Line 181 is to test not found case;
Line 182 is to test not found case when parameters are of multi types;
Line 183 is to test null in parameter which has >=1 index
I think maybe we should refer to Hive's field? In Hive, when not all arguments are numbers && not all arguments are strings, they are not compared as double.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95055050
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
+ val str1 = Literal("\u82b1\u82b1\u4e16\u754c")
+ val str2 = Literal("a")
+ val str3 = Literal("b")
+ val str4 = Literal("")
+ val str5 = Literal("999")
+ val strNull = Literal.create(null, StringType)
+
+ val bool1 = Literal(true)
+ val bool2 = Literal(false)
+
+ val int1 = Literal(1)
+ val int2 = Literal(2)
+ val int3 = Literal(3)
+ val int4 = Literal(999)
+ val intNull = Literal.create(null, IntegerType)
+
+ val double1 = Literal(1.221)
+ val double2 = Literal(1.222)
+ val double3 = Literal(1.224)
+
+ val timeStamp1 = Literal(new Timestamp(2016, 12, 27, 14, 22, 1, 1))
+ val timeStamp2 = Literal(new Timestamp(1988, 6, 3, 1, 1, 1, 1))
+ val timeStamp3 = Literal(new Timestamp(1990, 6, 5, 1, 1, 1, 1))
+
+ val date1 = Literal(new Date(1949, 1, 1))
+ val date2 = Literal(new Date(1979, 1, 1))
+ val date3 = Literal(new Date(1989, 1, 1))
+
+ checkEvaluation(Field(Seq(str1, str2, str3, str1)), 3)
+ checkEvaluation(Field(Seq(str2, str2, str2, str1)), 1)
+ checkEvaluation(Field(Seq(str4, str4, str4, str1)), 1)
--- End diff --
same as previous ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95720313
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala ---
@@ -17,11 +17,13 @@
package org.apache.spark.sql
+import java.sql.{Date, Timestamp}
+
import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.hadoop.mapreduce.lib.input.{TextInputFormat => NewTextInputFormat}
import org.scalatest.Matchers._
-import org.apache.spark.sql.catalyst.expressions.NamedExpression
+import org.apache.spark.sql.catalyst.expressions.{Literal, NamedExpression}
--- End diff --
Unused change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054508
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
--- End diff --
More examples please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95294259
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
+ _._1.dataType == children.head.dataType).map(_._2)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
--- End diff --
Got it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r94798265
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: Tuple2[Tuple2[ExprCode, DataType], Int]): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
+ s"""
+ ${code1}
+ else {
+ ${code2}
+ }
+ """
+ }
+
+ def dataTypeEqualsTarget(evalWithIndex: Tuple2[Tuple2[ExprCode, DataType], Int]): Boolean = {
--- End diff --
`def dataTypeEqualsTarget(evalWithIndex: ((ExprCode, DataType), Int)): Boolean`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on the issue:
https://github.com/apache/spark/pull/16476
@gczsjdy : I had one comment in past about `genIfElseStructure` but after giving more thought, I was not able to think about a better way to do that. I am fine withthe current version of code you have.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054547
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
--- End diff --
Could you fix the style, based on https://github.com/databricks/scala-style-guide#curly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95023996
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
--- End diff --
what's "case"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101770055
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
@viirya Maybe the order of `code` and `evalWithIndex` parameters should be changed.
@tejasapatil I agree with your opinion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72431/testReport)** for PR 16476 at commit [`b5deea1`](https://github.com/apache/spark/commit/b5deea17a7afd39061bcaa0756a28a0dfe292b76).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95080769
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
--- End diff --
`toList` probably causes performance overhead, I don't think we have to sacrifice the performance for using the pattern match. In the meantime, I still believe we don't have to check the data type during the runtime. It's supposed to be done during the `compile` time or only done once for the first time in `eval`.
The `Field` evaluation is quite confusing, as @gatorsmile suggested, we need to describe how to evaluate the value when sub expressions' data type are different.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
cc @chenghao-intel @adrian-wang
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101505199
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
Sorry I don't understand, how to use `foldLeft` approach? I think we can only use `foldRight` or `reduceRight`, because the code for latter children should be nested inner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r96348295
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
--- End diff --
Good idea, thx!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@cloud-fan Do you have comment on this version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #75038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75038/testReport)** for PR 16476 at commit [`9b4155a`](https://github.com/apache/spark/commit/9b4155a42796d4aa3224d109f28822196bf63041).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #74009 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74009/testReport)** for PR 16476 at commit [`dd598a4`](https://github.com/apache/spark/commit/dd598a40429493e5846009b33b0aa45043ee688a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101688198
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,105 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(
+ s"FIELD requires all arguments to be of AtomicType or explicitly indicating NULL")
+ } else {
+ TypeCheckResult.TypeCheckSuccess
+ }
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.length) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value)) {
+ dataTypeMatchIndex(index)
+ } else {
+ findEqual(index + 1)
+ }
+ }
+ }
+ if (target == null) 0 else findEqual(index = 0)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val dataTypes = children.map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, _), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
+ s"""
+ ${code1}
+ else {
--- End diff --
This is the floating `else`. As @tejasapatil said, looks weird.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r106334932
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,105 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
--- End diff --
If we try to cast all types to `DoubleType`, the test case for all `StringType` will fail.
While if we try to cast all types to `StringType`, parameters of '3' and '3.0' won't be determined as equal.
I have a solution to balance, we look at the 1st parameter, if it's of `NumericType`, we implicitly cast all parameters to `DoubleType`, else we cast all parameters to `StringType`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054575
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
+ val str1 = Literal("\u82b1\u82b1\u4e16\u754c")
+ val str2 = Literal("a")
+ val str3 = Literal("b")
+ val str4 = Literal("")
+ val str5 = Literal("999")
+ val strNull = Literal.create(null, StringType)
+
+ val bool1 = Literal(true)
+ val bool2 = Literal(false)
+
+ val int1 = Literal(1)
+ val int2 = Literal(2)
+ val int3 = Literal(3)
+ val int4 = Literal(999)
+ val intNull = Literal.create(null, IntegerType)
+
+ val double1 = Literal(1.221)
+ val double2 = Literal(1.222)
+ val double3 = Literal(1.224)
+
+ val timeStamp1 = Literal(new Timestamp(2016, 12, 27, 14, 22, 1, 1))
+ val timeStamp2 = Literal(new Timestamp(1988, 6, 3, 1, 1, 1, 1))
+ val timeStamp3 = Literal(new Timestamp(1990, 6, 5, 1, 1, 1, 1))
+
+ val date1 = Literal(new Date(1949, 1, 1))
+ val date2 = Literal(new Date(1979, 1, 1))
+ val date3 = Literal(new Date(1989, 1, 1))
+
+ checkEvaluation(Field(Seq(str1, str2, str3, str1)), 3)
+ checkEvaluation(Field(Seq(str2, str2, str2, str1)), 1)
+ checkEvaluation(Field(Seq(str4, str4, str4, str1)), 1)
+ checkEvaluation(Field(Seq(bool1, bool2, bool1, bool1)), 2)
+ checkEvaluation(Field(Seq(int1, int2, int3, int1)), 3)
+ checkEvaluation(Field(Seq(double2, double3, double1, double2)), 3)
+ checkEvaluation(Field(Seq(timeStamp1, timeStamp2, timeStamp3, timeStamp1)), 3)
+ checkEvaluation(Field(Seq(date1, date1, date2, date3)), 1)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int4)), 6)
+ checkEvaluation(Field(Seq(str5, str1, str2, str4)), 0)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(int1, strNull, intNull, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(strNull, int1, str1, str2, str3)), 0)
--- End diff --
This is to test `null`. Could you add the description?
```
If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r96347281
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala ---
@@ -17,11 +17,13 @@
package org.apache.spark.sql
+import java.sql.{Date, Timestamp}
--- End diff --
My bad.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72431/testReport)** for PR 16476 at commit [`b5deea1`](https://github.com/apache/spark/commit/b5deea17a7afd39061bcaa0756a28a0dfe292b76).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72427/testReport)** for PR 16476 at commit [`f2d16ec`](https://github.com/apache/spark/commit/f2d16ec73a38f75d968a39616761708c3b1ba735).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101515786
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
Oh, yeah, right, if use `foldLeft`, there is still a floating `else`. We can only use `foldRight` to remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95718465
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
--- End diff --
type: paramters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95281248
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
--- End diff --
`zip(Stream from 1)`, do we really need it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@gatorsmile Do you think we can merge this PR? Or is there something that need to modify?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95102842
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
--- End diff --
Yes, because in `checkEvaluation` function there is : `checkEvaluationWithGeneratedMutableProjection`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16476
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #74639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74639/testReport)** for PR 16476 at commit [`4e60b7c`](https://github.com/apache/spark/commit/4e60b7c52c0ca9e20296256607ce78741d80cea3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #74655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74655/testReport)** for PR 16476 at commit [`fc47e0f`](https://github.com/apache/spark/commit/fc47e0fbb8f196c91e69493d44a30a5b09a68f37).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72553/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93100/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16476
gentle ping @gczsjdy
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #71996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71996/testReport)** for PR 16476 at commit [`465f7a6`](https://github.com/apache/spark/commit/465f7a6fe7a3f94a8032283fe1ddc3a072b797f0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@tejasapatil Got it, thanks for your review. :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@viirya I have tested with Hive 2.1.1, `select field(2, '2', 2);` will return 2 while it will return 1 in MySQL, I also read Hive's code, where I found it will first compare the data type which is `Category` in Hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16476
It seems MySQL do need all arguments in `FIELD` have same type(or can be converted to same type): http://www.w3resource.com/mysql/string-functions/mysql-field-function.php
I think this is more reasonable, we don't have to follow hive strictly, can you check more database and see what's the most common way to evalaute `FIELD`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r97793794
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -20,8 +20,12 @@ package org.apache.spark.sql.catalyst.expressions
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.codegen._
+import org.apache.spark.sql.catalyst.util.TypeUtils
import org.apache.spark.sql.types._
+import scala.annotation.tailrec
--- End diff --
About the style. As I see, usually we put the import of scala packages above of spark packages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95055681
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
--- End diff --
At this point the code at lines 427-428 (ie. `.filter(dataTypeEqualsTarget)`) have ensured that this will always be true. The generated code will have this as `true` and you might as well get rid of the check here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101770992
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
Current code is ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@chenghao-intel
I think that the optimize rule will fold the parameters that have different types with param0, and then disorganize the parameters' index. Thanks.
@rxin
I have removed the [WIP] mark, this PR is ready now, could you please help me review it? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95732577
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
--- End diff --
Because Hive GenericUDFField supports primitive type(includes: boolean, byte, short, int...) & null.
If the types correspond to Spark SQL, they'll be AtomicType & NullType.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101690256
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
I think whats already present in the code is ok. Given that there is no better option without adding more complexity, lets stick with it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95078909
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1528,6 +1528,18 @@ object functions {
def factorial(e: Column): Column = withExpr { Factorial(e.expr) }
/**
+ * Returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ *
+ * @group normal_funcs
+ * @since 2.2.0
+ */
+ @scala.annotation.varargs
+ def field(expr1: Column, expr2: Column, exprs: Column*): Column = withExpr {
--- End diff --
Sure, I have removed this.
Still curious, is it inappropriate to be in Dataset API?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95080738
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
Maybe I can change this function's name? But actually I can't think of a better name. : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95080295
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
--- End diff --
@gatorsmile Actually, the case:
`checkAnswer(testData.selectExpr("field('\u82b1\u82b1\u4e16\u754c', 'a', 1.23, true, '\u82b1\u82b1\u4e16\u754c')"), Row(4))`
will produce a child: Seq[Expression] an ArrayBuffer, it's not a list, so can't use head::tail.
So there are 2 ways:
1. remove the `toList` and do another pattern match for ArrayBuffer, which I think is not neat.
2. keep the `toList`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72489/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101436180
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
--- End diff --
Oh. I mean the error message `s"FIELD requires all arguments to be of AtomicType or NullType"`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95079031
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
+ val str1 = Literal("\u82b1\u82b1\u4e16\u754c")
+ val str2 = Literal("a")
+ val str3 = Literal("b")
+ val str4 = Literal("")
+ val str5 = Literal("999")
+ val strNull = Literal.create(null, StringType)
+
+ val bool1 = Literal(true)
+ val bool2 = Literal(false)
+
+ val int1 = Literal(1)
+ val int2 = Literal(2)
+ val int3 = Literal(3)
+ val int4 = Literal(999)
+ val intNull = Literal.create(null, IntegerType)
+
+ val double1 = Literal(1.221)
+ val double2 = Literal(1.222)
+ val double3 = Literal(1.224)
+
+ val timeStamp1 = Literal(new Timestamp(2016, 12, 27, 14, 22, 1, 1))
+ val timeStamp2 = Literal(new Timestamp(1988, 6, 3, 1, 1, 1, 1))
+ val timeStamp3 = Literal(new Timestamp(1990, 6, 5, 1, 1, 1, 1))
+
+ val date1 = Literal(new Date(1949, 1, 1))
+ val date2 = Literal(new Date(1979, 1, 1))
+ val date3 = Literal(new Date(1989, 1, 1))
+
+ checkEvaluation(Field(Seq(str1, str2, str3, str1)), 3)
+ checkEvaluation(Field(Seq(str2, str2, str2, str1)), 1)
+ checkEvaluation(Field(Seq(str4, str4, str4, str1)), 1)
+ checkEvaluation(Field(Seq(bool1, bool2, bool1, bool1)), 2)
+ checkEvaluation(Field(Seq(int1, int2, int3, int1)), 3)
+ checkEvaluation(Field(Seq(double2, double3, double1, double2)), 3)
+ checkEvaluation(Field(Seq(timeStamp1, timeStamp2, timeStamp3, timeStamp1)), 3)
+ checkEvaluation(Field(Seq(date1, date1, date2, date3)), 1)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int4)), 6)
+ checkEvaluation(Field(Seq(str5, str1, str2, str4)), 0)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(int1, strNull, intNull, bool1, date1, timeStamp2, int3)), 0)
--- End diff --
I have removed line 181, since line 182 actually covers not found case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@cloud-fan Could you please help me review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101688665
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
--- End diff --
I think it looks like:
${evalChildren.zip(dataTypes).zipWithIndex.tail.filter { x =>
dataTypeMatchIndex.contains(x._2)
}.foldRight("") { (code: String, evalWithIndex: ((ExprCode, DataType), Int)) =>
val ((eval, _), index) = evalWithIndex
val condition = ctx.genEqual(targetDataType, eval.value, target.value)
s"""
${eval.code}
if ($condition) {
${ev.value} = ${index};
} else {
$code
}
"""
}
You can do this with a function like you did before. It will have a empty "else" block at the end.
However this doesn't affect the functionality, just dealing with how the code looks. I don't have strong option about this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95720884
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
--- End diff --
Any reason we can only support the two types?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95080333
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
--- End diff --
@tejasapatil Actually, I think it's tail recursion, so the compiler will do the optimization, then it has the same performance with iteration edition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95281159
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
+ _._1.dataType == children.head.dataType).map(_._2)
--- End diff --
`_._1.dataType.sameType(children.head.dataType)`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72427/testReport)** for PR 16476 at commit [`f2d16ec`](https://github.com/apache/spark/commit/f2d16ec73a38f75d968a39616761708c3b1ba735).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95055160
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
--- End diff --
Do your unit tests cover generated code ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r102671021
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,105 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
--- End diff --
For the current implementation, I didn't do implicit cast to in accord with Hive.
For example:
`select field(3, '3', 3);` will return 2 in Hive, while it will return 1 if the implicit cast's done.
So I think maybe it's better not to do the implicit cast, and to allow that children have different types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #93594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93594/testReport)** for PR 16476 at commit [`c75d786`](https://github.com/apache/spark/commit/c75d786aef5632098a78be3640a67463374e9c8b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72427/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@maropu Sure, I will update it this week.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16476
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95080582
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
--- End diff --
you can add the annotation `@tailrec` for explicitly declare that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72994/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101478064
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
--- End diff --
Looks good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101436701
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,104 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else {
+ TypeCheckResult.TypeCheckSuccess
+ }
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.length) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value)) {
+ dataTypeMatchIndex(index)
+ } else {
+ findEqual(index + 1)
+ }
+ }
+ }
+ if (target == null) 0 else findEqual(index = 0)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
--- End diff --
nit: We may not need to genCode for the children with the data types different than the target.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r94797482
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
--- End diff --
Nit: one more space before `findEqual`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95282465
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
+ _._1.dataType == children.head.dataType).map(_._2)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.size) {
--- End diff --
if `dataTypeMatchIndex` is `Array[Int]`, then we'd better use `dataTypeMatchIndex.length` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72994 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72994/testReport)** for PR 16476 at commit [`0665a1d`](https://github.com/apache/spark/commit/0665a1d0d76bdb553010775eb1c53b8d461e792a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101468470
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,104 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else {
+ TypeCheckResult.TypeCheckSuccess
+ }
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.length) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value)) {
+ dataTypeMatchIndex(index)
+ } else {
+ findEqual(index + 1)
+ }
+ }
+ }
+ if (target == null) 0 else findEqual(index = 0)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
--- End diff --
There is a filter function on line 441 to avoid genCode for type mismatch children
`filter(x => dataTypeMatchIndex.contains(x._2))`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r106157856
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,105 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
--- End diff --
I think I should add an interface(and`Field` mixin that) like `ImplicitCastInputTypes`, the difference is that, the new interface casts all parameter to the same type(at least, they can be cast to string type)\u3002
In the new interface, I fist try to cast all parameters to NumericType(if don't do this, s"2.3" won't be determined equal to s"2.30"), if that fails, I cast all parameters to StringType.
Do you think this is a good idea? @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95720330
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala ---
@@ -17,11 +17,13 @@
package org.apache.spark.sql
+import java.sql.{Date, Timestamp}
--- End diff --
Unused change too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95023960
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1528,6 +1528,18 @@ object functions {
def factorial(e: Column): Column = withExpr { Factorial(e.expr) }
/**
+ * Returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ *
+ * @group normal_funcs
+ * @since 2.2.0
+ */
+ @scala.annotation.varargs
+ def field(expr1: Column, expr2: Column, exprs: Column*): Column = withExpr {
--- End diff --
can we remove this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95060307
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
+ val str1 = Literal("\u82b1\u82b1\u4e16\u754c")
+ val str2 = Literal("a")
+ val str3 = Literal("b")
+ val str4 = Literal("")
+ val str5 = Literal("999")
+ val strNull = Literal.create(null, StringType)
+
+ val bool1 = Literal(true)
+ val bool2 = Literal(false)
+
+ val int1 = Literal(1)
+ val int2 = Literal(2)
+ val int3 = Literal(3)
+ val int4 = Literal(999)
+ val intNull = Literal.create(null, IntegerType)
+
+ val double1 = Literal(1.221)
+ val double2 = Literal(1.222)
+ val double3 = Literal(1.224)
+
+ val timeStamp1 = Literal(new Timestamp(2016, 12, 27, 14, 22, 1, 1))
+ val timeStamp2 = Literal(new Timestamp(1988, 6, 3, 1, 1, 1, 1))
+ val timeStamp3 = Literal(new Timestamp(1990, 6, 5, 1, 1, 1, 1))
+
+ val date1 = Literal(new Date(1949, 1, 1))
+ val date2 = Literal(new Date(1979, 1, 1))
+ val date3 = Literal(new Date(1989, 1, 1))
+
+ checkEvaluation(Field(Seq(str1, str2, str3, str1)), 3)
+ checkEvaluation(Field(Seq(str2, str2, str2, str1)), 1)
+ checkEvaluation(Field(Seq(str4, str4, str4, str1)), 1)
+ checkEvaluation(Field(Seq(bool1, bool2, bool1, bool1)), 2)
+ checkEvaluation(Field(Seq(int1, int2, int3, int1)), 3)
+ checkEvaluation(Field(Seq(double2, double3, double1, double2)), 3)
+ checkEvaluation(Field(Seq(timeStamp1, timeStamp2, timeStamp3, timeStamp1)), 3)
+ checkEvaluation(Field(Seq(date1, date1, date2, date3)), 1)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int4)), 6)
+ checkEvaluation(Field(Seq(str5, str1, str2, str4)), 0)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(int1, strNull, intNull, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(strNull, int1, str1, str2, str3)), 0)
--- End diff --
@gatorsmile Sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95281046
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
--- End diff --
Array[Int] instead? Seq[Int] probably a LinkedList in its concrete implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95080770
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
+ s"""
+ ${code1}
+ else {
+ ${code2}
+ }
+ """
+ }
+
+ def dataTypeEqualsTarget(evalWithIndex: ((ExprCode, DataType), Int)): Boolean = {
+ val ((eval, dataType), index) = evalWithIndex
+ dataType.equals(targetDataType)
+ }
+
+ ev.copy(code = s"""
+ ${target.code}
+ boolean ${ev.isNull} = false;
+ int ${ev.value} = 0;
+ ${rest.zip(restDataType).zipWithIndex.map(x => (x._1, x._2 + 1)).filter(
--- End diff --
Cool, thx.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@cloud-fan I have submitted a new version to support implicit cast.
We determine the implicit cast type in analysis stage, so maybe we won't do `eval` then, so we can't be 100% intelligent, for example to cast parameters of `("2", 2, "a", "2")` to `(2, 2, somewhat, 2)`.
I made a trade off here to follow the type of the first parameter(the target) in general.
The specific type rules here:
1. If all the parameters are of the same type, we don't do implicit cast.
2. If the first parameter is of `NumericType`, we implicit cast all other parameters to `NumericType` (only in this way, "2.0" and "2.00" can be treated equal)and do the comparison, those that can't be cast to `NumericType` will be set to null of `DoubleType`.
3. else, we implicit cast all other parameters to `StringType`, this can always be successful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71996/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95055535
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
+ val str1 = Literal("\u82b1\u82b1\u4e16\u754c")
+ val str2 = Literal("a")
+ val str3 = Literal("b")
+ val str4 = Literal("")
+ val str5 = Literal("999")
+ val strNull = Literal.create(null, StringType)
+
+ val bool1 = Literal(true)
+ val bool2 = Literal(false)
+
+ val int1 = Literal(1)
+ val int2 = Literal(2)
+ val int3 = Literal(3)
+ val int4 = Literal(999)
+ val intNull = Literal.create(null, IntegerType)
+
+ val double1 = Literal(1.221)
+ val double2 = Literal(1.222)
+ val double3 = Literal(1.224)
+
+ val timeStamp1 = Literal(new Timestamp(2016, 12, 27, 14, 22, 1, 1))
+ val timeStamp2 = Literal(new Timestamp(1988, 6, 3, 1, 1, 1, 1))
+ val timeStamp3 = Literal(new Timestamp(1990, 6, 5, 1, 1, 1, 1))
+
+ val date1 = Literal(new Date(1949, 1, 1))
+ val date2 = Literal(new Date(1979, 1, 1))
+ val date3 = Literal(new Date(1989, 1, 1))
+
+ checkEvaluation(Field(Seq(str1, str2, str3, str1)), 3)
+ checkEvaluation(Field(Seq(str2, str2, str2, str1)), 1)
+ checkEvaluation(Field(Seq(str4, str4, str4, str1)), 1)
+ checkEvaluation(Field(Seq(bool1, bool2, bool1, bool1)), 2)
+ checkEvaluation(Field(Seq(int1, int2, int3, int1)), 3)
+ checkEvaluation(Field(Seq(double2, double3, double1, double2)), 3)
+ checkEvaluation(Field(Seq(timeStamp1, timeStamp2, timeStamp3, timeStamp1)), 3)
+ checkEvaluation(Field(Seq(date1, date1, date2, date3)), 1)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int4)), 6)
+ checkEvaluation(Field(Seq(str5, str1, str2, str4)), 0)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(int1, strNull, intNull, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(strNull, int1, str1, str2, str3)), 0)
--- End diff --
This fails with the patch.
hc.sql("""SELECT FIELD("tejas", 34, "patil", true, null, "tejas") FROM src LIMIT 1""").collect.foreach(println)
Removing `null` makes it work. Can you check on your side ? It worked with Hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054951
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
--- End diff --
This UDF accepts any mix atomic types so one can even get fancy with the inputs. Would recommend mentioning that in the doc (given that you have tests for that below)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the issue:
https://github.com/apache/spark/pull/16476
@gczsjdy can you please add [WIP] in the title, until you feel the code is ready for review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95294460
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
--- End diff --
Actually, it's short for `zipWithIndex.map(x => (x._1, x._2 + 1))`.
I realized it makes people confused, and have changed it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72505/testReport)** for PR 16476 at commit [`66e779e`](https://github.com/apache/spark/commit/66e779efd8aeb0027427577fbe225893e47bd419).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054704
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
--- End diff --
Can we avoid `toList` for each recursive call?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95060664
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
--- End diff --
I can give 2 examples, one for Integer, one for String.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95719439
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
--- End diff --
This description "comparing will be done based on type firstly" looks unclear. Can you rephrase it a bit?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r96347397
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
--- End diff --
That's for user's explicit indication of NULL, that's legal in Hive's `field` expression.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74009/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74655/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101766864
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,104 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else {
+ TypeCheckResult.TypeCheckSuccess
+ }
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.length) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value)) {
+ dataTypeMatchIndex(index)
+ } else {
+ findEqual(index + 1)
+ }
+ }
+ }
+ if (target == null) 0 else findEqual(index = 0)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
--- End diff --
You mean do not gen any code, not calling `genCode` at all?
I think maybe we can't modify the upper level code for this reason. Is there a better solution?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #93100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93100/testReport)** for PR 16476 at commit [`4b63e94`](https://github.com/apache/spark/commit/4b63e945739ac0c9d2dc572354edc97e2025683e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
ping @rxin @gatorsmile @tejasapatil @viirya
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95060369
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
+ val str1 = Literal("\u82b1\u82b1\u4e16\u754c")
+ val str2 = Literal("a")
+ val str3 = Literal("b")
+ val str4 = Literal("")
+ val str5 = Literal("999")
+ val strNull = Literal.create(null, StringType)
+
+ val bool1 = Literal(true)
+ val bool2 = Literal(false)
+
+ val int1 = Literal(1)
+ val int2 = Literal(2)
+ val int3 = Literal(3)
+ val int4 = Literal(999)
+ val intNull = Literal.create(null, IntegerType)
+
+ val double1 = Literal(1.221)
+ val double2 = Literal(1.222)
+ val double3 = Literal(1.224)
+
+ val timeStamp1 = Literal(new Timestamp(2016, 12, 27, 14, 22, 1, 1))
+ val timeStamp2 = Literal(new Timestamp(1988, 6, 3, 1, 1, 1, 1))
+ val timeStamp3 = Literal(new Timestamp(1990, 6, 5, 1, 1, 1, 1))
+
+ val date1 = Literal(new Date(1949, 1, 1))
+ val date2 = Literal(new Date(1979, 1, 1))
+ val date3 = Literal(new Date(1989, 1, 1))
+
+ checkEvaluation(Field(Seq(str1, str2, str3, str1)), 3)
+ checkEvaluation(Field(Seq(str2, str2, str2, str1)), 1)
+ checkEvaluation(Field(Seq(str4, str4, str4, str1)), 1)
+ checkEvaluation(Field(Seq(bool1, bool2, bool1, bool1)), 2)
+ checkEvaluation(Field(Seq(int1, int2, int3, int1)), 3)
+ checkEvaluation(Field(Seq(double2, double3, double1, double2)), 3)
+ checkEvaluation(Field(Seq(timeStamp1, timeStamp2, timeStamp3, timeStamp1)), 3)
+ checkEvaluation(Field(Seq(date1, date1, date2, date3)), 1)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int4)), 6)
+ checkEvaluation(Field(Seq(str5, str1, str2, str4)), 0)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(int1, strNull, intNull, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(strNull, int1, str1, str2, str3)), 0)
--- End diff --
@tejasapatil It doesn't work with my code because it only support AtomicType. While user's writing of 'null' is NullType, I will add NullType support to be consistent with Hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101466954
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
--- End diff --
You are right, but is it better to be `FIELD requires all arguments to be of AtomicType or explicitly indicating NULL` ? Because `NullType` might be confusing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95719293
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
--- End diff --
" or NullType"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL][WIP] Implement expression field
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the issue:
https://github.com/apache/spark/pull/16476
Since the different data type will be simply ignored, I think we'd better also add the optimization rule in `Optimizer`.
As well as the python/scala API support, but need to confirm with @rxin, why we don't need the API `field`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054651
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
--- End diff --
`findEqual(target, children.tail, index=1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95283107
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1528,6 +1528,18 @@ object functions {
def factorial(e: Column): Column = withExpr { Factorial(e.expr) }
/**
+ * Returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ *
+ * @group normal_funcs
+ * @since 2.2.0
+ */
+ @scala.annotation.varargs
+ def field(expr1: Column, expr2: Column, exprs: Column*): Column = withExpr {
--- End diff --
@rxin can you explain a little bit why we remove this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@gatorsmile Hi, this patch has passed all tests, is there some code I still need to modify? Thank you for working on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72505/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r101770685
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,104 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else {
+ TypeCheckResult.TypeCheckSuccess
+ }
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.length) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value)) {
+ dataTypeMatchIndex(index)
+ } else {
+ findEqual(index + 1)
+ }
+ }
+ }
+ if (target == null) 0 else findEqual(index = 0)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
--- End diff --
I meant just genCode for the children with the same data type. genCode should be lightweight, so it is not a strong option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r97796700
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,99 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.length) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value)) {
+ dataTypeMatchIndex(index)
+ } else {
+ findEqual(index + 1)
+ }
+ }
+ }
+ if (target == null) 0 else findEqual(index = 0)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val dataTypes = children.map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, _), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
+ s"""
+ ${code1}
+ else {
+ ${code2}
+ }
+ """
+ }
+
+ ev.copy(code = s"""
+ ${target.code}
--- End diff --
If `target.value == null`, we don't need to evaluate the expressions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95055087
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
--- End diff --
I feel like you have comprehensive testing but at the same time feels like there is overlap amongst the tests in terms of coverage. There is room of reducing the tests while still having same coverage. eg you don't need 5 strings and use lesser.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72489/testReport)** for PR 16476 at commit [`a8d22e3`](https://github.com/apache/spark/commit/a8d22e382e07b1bc6acccded13648ab0d22ed255).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054605
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
+ val str1 = Literal("\u82b1\u82b1\u4e16\u754c")
+ val str2 = Literal("a")
+ val str3 = Literal("b")
+ val str4 = Literal("")
+ val str5 = Literal("999")
+ val strNull = Literal.create(null, StringType)
+
+ val bool1 = Literal(true)
+ val bool2 = Literal(false)
+
+ val int1 = Literal(1)
+ val int2 = Literal(2)
+ val int3 = Literal(3)
+ val int4 = Literal(999)
+ val intNull = Literal.create(null, IntegerType)
+
+ val double1 = Literal(1.221)
+ val double2 = Literal(1.222)
+ val double3 = Literal(1.224)
+
+ val timeStamp1 = Literal(new Timestamp(2016, 12, 27, 14, 22, 1, 1))
+ val timeStamp2 = Literal(new Timestamp(1988, 6, 3, 1, 1, 1, 1))
+ val timeStamp3 = Literal(new Timestamp(1990, 6, 5, 1, 1, 1, 1))
+
+ val date1 = Literal(new Date(1949, 1, 1))
+ val date2 = Literal(new Date(1979, 1, 1))
+ val date3 = Literal(new Date(1989, 1, 1))
+
+ checkEvaluation(Field(Seq(str1, str2, str3, str1)), 3)
+ checkEvaluation(Field(Seq(str2, str2, str2, str1)), 1)
+ checkEvaluation(Field(Seq(str4, str4, str4, str1)), 1)
+ checkEvaluation(Field(Seq(bool1, bool2, bool1, bool1)), 2)
+ checkEvaluation(Field(Seq(int1, int2, int3, int1)), 3)
+ checkEvaluation(Field(Seq(double2, double3, double1, double2)), 3)
+ checkEvaluation(Field(Seq(timeStamp1, timeStamp2, timeStamp3, timeStamp1)), 3)
+ checkEvaluation(Field(Seq(date1, date1, date2, date3)), 1)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int4)), 6)
+ checkEvaluation(Field(Seq(str5, str1, str2, str4)), 0)
+ checkEvaluation(Field(Seq(int4, double3, str5, bool1, date1, timeStamp2, int3)), 0)
+ checkEvaluation(Field(Seq(int1, strNull, intNull, bool1, date1, timeStamp2, int3)), 0)
--- End diff --
What is the purpose of these checks?
Based on MySQL's `field` function, the type casting rules is described as
```
If all arguments to FIELD() are strings, all arguments are compared as strings. If all arguments are numbers, they are compared as numbers. Otherwise, the arguments are compared as double.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95060633
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
--- End diff --
Yeah, it's more reasonable to use expr(n), thx.
Probably it should be AtomicType or NullType to support user's writing of null.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95282681
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
+ _._1.dataType == children.head.dataType).map(_._2)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ @tailrec def findEqual(index: Int): Int = {
+ if (index == dataTypeMatchIndex.size) {
+ 0
+ } else {
+ val value = children(dataTypeMatchIndex(index)).eval(input)
+ if (value != null && ordering.equiv(target, value))
--- End diff --
use braces. see https://github.com/databricks/scala-style-guide#curly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95023939
--- Diff: python/pyspark/sql/functions.py ---
@@ -320,6 +320,22 @@ def countDistinct(col, *cols):
return Column(jc)
+@since(2.2)
+def field(*cols):
--- End diff --
can we remove this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95292481
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
--- End diff --
Won't it be inconsistent with children? Since children's type is Seq[Expression]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95060448
--- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ConditionalExpressionSuite.scala ---
@@ -137,4 +139,48 @@ class ConditionalExpressionSuite extends SparkFunSuite with ExpressionEvalHelper
checkEvaluation(CaseKeyWhen(c6, Seq(c5, c2, c4, c3)), null, row)
checkEvaluation(CaseKeyWhen(literalNull, Seq(c2, c5, c1, c6)), null, row)
}
+
+ test("case field") {
--- End diff --
@rxin My bad, will take it out
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16476
I don't think we should follow Hive for everything even for mistakes. I think we should not care about which one is more popular, but which one is more reasonable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
Thanks @viirya for the review and sorry for the late reply.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16476
@tejasapatil Sorry to bother, is there still something need to modify?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75038/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95055899
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(_.dataType.isInstanceOf[AtomicType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
+ def findEqual(target: Any, params: Seq[Expression], index: Int): Int = {
+ params.toList match {
+ case Nil => 0
+ case head::tail if targetDataType == head.dataType
+ && head.eval(input) != null && ordering.equiv(target, head.eval(input)) => index
+ case _ => findEqual(target, params.tail, index + 1)
+ }
+ }
+ if(target == null)
+ 0
+ else
+ findEqual(target, children.tail, 1)
+ }
+
+ protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ val evalChildren = children.map(_.genCode(ctx))
+ val target = evalChildren(0)
+ val targetDataType = children(0).dataType
+ val rest = evalChildren.drop(1)
+ val restDataType = children.drop(1).map(_.dataType)
+
+ def updateEval(evalWithIndex: ((ExprCode, DataType), Int)): String = {
+ val ((eval, dataType), index) = evalWithIndex
+ s"""
+ ${eval.code}
+ if (${dataType.equals(targetDataType)}
+ && ${ctx.genEqual(targetDataType, eval.value, target.value)}) {
+ ${ev.value} = ${index};
+ }
+ """
+ }
+
+ def genIfElseStructure(code1: String, code2: String): String = {
+ s"""
+ ${code1}
+ else {
+ ${code2}
+ }
+ """
+ }
+
+ def dataTypeEqualsTarget(evalWithIndex: ((ExprCode, DataType), Int)): Boolean = {
+ val ((eval, dataType), index) = evalWithIndex
+ dataType.equals(targetDataType)
+ }
+
+ ev.copy(code = s"""
+ ${target.code}
+ boolean ${ev.isNull} = false;
+ int ${ev.value} = 0;
+ ${rest.zip(restDataType).zipWithIndex.map(x => (x._1, x._2 + 1)).filter(
--- End diff --
`.zipWithIndex.map(x => (x._1, x._2 + 1))` can be simplified as `.zip(Stream from 1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95721703
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
--- End diff --
I think it is better to move this sentence to below the next line. I.e.,
It's also acceptable to give parameters of different types. When the parameters have different types, ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72553/testReport)** for PR 16476 at commit [`d22dbb3`](https://github.com/apache/spark/commit/d22dbb345d7d48bae9a69e09d4049be58df6af61).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #71996 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71996/testReport)** for PR 16476 at commit [`465f7a6`](https://github.com/apache/spark/commit/465f7a6fe7a3f94a8032283fe1ddc3a072b797f0).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r102616898
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,105 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
--- End diff --
we need a rule in `TypeCoercion` to make sure all children are of same type(after implicit cast maybe?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72554/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054878
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
--- End diff --
change `str` to `expr` here as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95054387
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters' types should be subtypes of AtomicType.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(str, str1, str2, ...) - Returns the index of str in the str1,str2,... or 0 if not found.",
--- End diff --
Can we use `expr1, expr2, expr3` here? The type can be any atomic type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r94796481
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +341,91 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of str in (str1, str2, ...) list or 0 if not found.
--- End diff --
Nit: delete a space before *
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL][WIP] Implement expression fiel...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r95282270
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,102 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types.
+ * If the search string is NULL, the return value is 0 because NULL fails equality comparison with any value.
+ * When the paramters have different types, comparing will be done based on type firstly,
+ * for example, ''999'' won't be considered equal with 999, no implicit cast will be done here.
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Seq[Int] = children.tail.zip(Stream from 1).filter(
+ _._1.dataType == children.head.dataType).map(_._2)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ if (children.length <= 1) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires at least 2 arguments")
+ } else if (!children.forall(
+ e => e.dataType.isInstanceOf[AtomicType] || e.dataType.isInstanceOf[NullType])) {
+ TypeCheckResult.TypeCheckFailure(s"FIELD requires all arguments to be of AtomicType")
+ } else
+ TypeCheckResult.TypeCheckSuccess
+ }
+
+ override def dataType: DataType = IntegerType
+ override def eval(input: InternalRow): Any = {
+ val target = children.head.eval(input)
+ val targetDataType = children.head.dataType
--- End diff --
Unused code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16476
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r99753353
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -20,8 +20,12 @@ package org.apache.spark.sql.catalyst.expressions
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.expressions.codegen._
+import org.apache.spark.sql.catalyst.util.TypeUtils
import org.apache.spark.sql.types._
+import scala.annotation.tailrec
--- End diff --
Sorry, it's my fault.
I will run dev/scalastyle first before I push my changes next time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16476: [SPARK-19084][SQL] Implement expression field
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16476
**[Test build #72554 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72554/testReport)** for PR 16476 at commit [`c23deb5`](https://github.com/apache/spark/commit/c23deb579d3ff0d3bb432484c2a8cce487bb2fa2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r106334083
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +343,105 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
+ * It's also acceptable to give parameters of different types. When the parameters have different
+ * types, comparing will be done based on type firstly. For example, ''999'' 's type is StringType,
+ * while 999's type is IntegerType, so that no further comparison need to be done since they have
+ * different types.
+ * If the search expression is NULL, the return value is 0 because NULL fails equality comparison
+ * with any value.
+ * To also point out, no implicit cast will be done in this expression.
+ */
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(expr, expr1, expr2, ...) - Returns the index of expr in the expr1, expr2, ... or 0 if not found.",
+ extended = """
+ Examples:
+ > SELECT _FUNC_(10, 9, 3, 10, 4);
+ 3
+ > SELECT _FUNC_('a', 'b', 'c', 'd', 'a');
+ 4
+ > SELECT _FUNC_('999', 'a', 999, 9.99, '999');
+ 4
+ """)
+// scalastyle:on line.size.limit
+case class Field(children: Seq[Expression]) extends Expression {
+
+ /** Even if expr is not found in (expr1, expr2, ...) list, the value will be 0, not null */
+ override def nullable: Boolean = false
+ override def foldable: Boolean = children.forall(_.foldable)
+
+ private lazy val ordering = TypeUtils.getInterpretedOrdering(children(0).dataType)
+
+ private val dataTypeMatchIndex: Array[Int] = children.zipWithIndex.tail.filter(
+ _._1.dataType.sameType(children.head.dataType)).map(_._2).toArray
+
+ override def checkInputDataTypes(): TypeCheckResult = {
--- End diff --
I met a problem, if I first try to cast all parameters to DoubleType in `TypeCoercion`(as described in 2nd paragraph last comment). I should really cast in order to know it can success or not, but that will 'execute' in analysis stage, that seems not right.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16476: [SPARK-19084][SQL] Implement expression field
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on a diff in the pull request:
https://github.com/apache/spark/pull/16476#discussion_r96347166
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala ---
@@ -340,3 +344,96 @@ object CaseKeyWhen {
CaseWhen(cases, elseValue)
}
}
+
+/**
+ * A function that returns the index of expr in (expr1, expr2, ...) list or 0 if not found.
+ * It takes at least 2 parameters, and all parameters should be subtype of AtomicType or NullType.
--- End diff --
Yes, that's right.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org