You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2015/10/25 09:05:58 UTC
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/9270
[SPARK-9162][SQL] Implement code generation for ScalaUDF
JIRA: https://issues.apache.org/jira/browse/SPARK-9162
Currently ScalaUDF extends CodegenFallback and doesn't provide code generation implementation. This path implements code generation for ScalaUDF.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 scalaudf-codegen
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9270.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9270
----
commit 5e8efeacdf35df7281224338866a9b18207fd27f
Author: Liang-Chi Hsieh <vi...@appier.com>
Date: 2015-10-25T08:00:31Z
Add codegen support for ScalaUDF.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-150900046
**[Test build #44315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44315/consoleFull)** for PR 9270 at commit [`5e8efea`](https://github.com/apache/spark/commit/5e8efeacdf35df7281224338866a9b18207fd27f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153270890
**[Test build #44893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44893/consoleFull)** for PR 9270 at commit [`4a1ec67`](https://github.com/apache/spark/commit/4a1ec678a4ed51b4644ab614c25ad7a3e832b515).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:\n * ` case class RegisteredExecutor(hostname: String) extends CoarseGrainedClusterMessage`\n * `public class JavaIsotonicRegressionExample `\n * `public class JavaNaiveBayesExample `\n * `case class UnresolvedStar(target: Option[Seq[String]]) extends Star with Unevaluable `\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154328910
**[Test build #45199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45199/consoleFull)** for PR 9270 at commit [`b355895`](https://github.com/apache/spark/commit/b3558957c7269ded7e5e90822dcd96c097651828).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153272176
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153268456
**[Test build #44893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44893/consoleFull)** for PR 9270 at commit [`4a1ec67`](https://github.com/apache/spark/commit/4a1ec678a4ed51b4644ab614c25ad7a3e832b515).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943320
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
What about function?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943091
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
For that one, can you just do callFunc.getClass.getName?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153272325
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42962687
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
Looks like `function.getClass.getName` will cause other problem (`java.lang.IncompatibleClassChangeError`) as the following failed tests show.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153295647
**[Test build #44898 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44898/consoleFull)** for PR 9270 at commit [`6b94a38`](https://github.com/apache/spark/commit/6b94a380c6d870bd93572e32168c2a3892dfa76f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151059691
**[Test build #44343 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44343/consoleFull)** for PR 9270 at commit [`a9f31c9`](https://github.com/apache/spark/commit/a9f31c9f74a0b7458ed029b121f8b9bbd8fee93a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42942985
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
Maybe I'm missing something, but I think you can just write a loop instead of having branches?
```
...
val funcClassName = callFunc.getClass.getName
...
val evals = children.map(_.gen(ctx))
...
// generate callFunc
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153273390
@davies May I ask what the data types you meant? scala.Function0 to scala.Function22?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42942936
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
You meant using a script to generate it like the `f`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-150911095
**[Test build #44315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44315/consoleFull)** for PR 9270 at commit [`5e8efea`](https://github.com/apache/spark/commit/5e8efeacdf35df7281224338866a9b18207fd27f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154345120
**[Test build #45199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45199/consoleFull)** for PR 9270 at commit [`b355895`](https://github.com/apache/spark/commit/b3558957c7269ded7e5e90822dcd96c097651828).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `final class ShuffleSortDataFormat extends SortDataFormat<PackedRecordPointer, LongArray> `\n * `final class UnsafeSortDataFormat extends SortDataFormat<RecordPointerAndKeyPrefix, LongArray> `\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151041697
**[Test build #44343 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44343/consoleFull)** for PR 9270 at commit [`a9f31c9`](https://github.com/apache/spark/commit/a9f31c9f74a0b7458ed029b121f8b9bbd8fee93a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153275269
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154250202
@davies yes. I will do it soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r44117245
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -965,6 +969,80 @@ case class ScalaUDF(
}
// scalastyle:on
+
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter")
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName" +
+ s".createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)" +
+ s"expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName" +
+ s".createToCatalystConverter((($scalaUDFClassName)expressions" +
+ s"[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ // This must be called before children expressions' codegen
+ // because ctx.references is used in genCodeForConverter
--- End diff --
I've added that. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9270
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943031
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
Hmm, part of this can be reduced like you show. But seems we still have a (smaller) pattern matching. I will do it later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42942860
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
can we put these branches in a loop?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r43721230
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -965,6 +969,80 @@ case class ScalaUDF(
}
// scalastyle:on
+
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter")
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName" +
+ s".createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)" +
+ s"expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName" +
+ s".createToCatalystConverter((($scalaUDFClassName)expressions" +
+ s"[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ // This must be called before children expressions' codegen
+ // because ctx.references is used in genCodeForConverter
--- End diff --
It's better to save the index when you push the expression into `ctx.references`, otherwise it's easy to be wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151040790
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943369
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
If you meant `function: AnyRef` in `ScalaUDF`, it is the user-defined function. `function.getClass.getName` will get something like ` org.apache.spark.sql.UDFSuite$$anonfun$10$$anonfun$apply$mcV$sp$12` in `UDFSuite`. So I think it should not work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943394
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
Why wouldn't it work? Isn't that better because we can even specialize it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153293878
**[Test build #44897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44897/consoleFull)** for PR 9270 at commit [`abb3750`](https://github.com/apache/spark/commit/abb375055b700c1cc431d093ed11ea536be44ce0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `case class UnresolvedStar(target: Option[Seq[String]]) extends Star with Unevaluable `\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151031386
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42962733
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
But `classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName` works. So I will revert to it then.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153267797
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154345230
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154345232
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45199/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154448615
@viirya This patch looks good to me now. Another thing is that we can generate code for catalyst converter now (see RowEncoder), and we have a plan to replace catalyst converter with the generated one. Do you want to try it in this PR or leave that as a follow up one?
cc @marmbrus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153270914
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44893/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-150911127
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42959109
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
As I tried, using `function.getClass.getName` causes `java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Incompatible return types` in `ZeroArgument UDF` test.
But other tests are passed. So I add a condition to take care of zero argument function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153275336
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153276189
**[Test build #44898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44898/consoleFull)** for PR 9270 at commit [`6b94a38`](https://github.com/apache/spark/commit/6b94a380c6d870bd93572e32168c2a3892dfa76f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153295865
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44898/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-150899670
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r43659293
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,122 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ // This must be called before children expressions' codegen
+ // because ctx.references is used in genCodeForConverter
+ val converterTerms = (0 until children.size).map(genCodeForConverter(ctx, _))
+
+ // Initialize user-defined function
+ val funcClassName = children.size match {
+ case 0 =>
+ classOf[() => Any].getName
+ case 1 =>
+ classOf[(Any) => Any].getName
+ case 2 =>
+ classOf[(Any, Any) => Any].getName
+ case 3 =>
+ classOf[(Any, Any, Any) => Any].getName
+ case 4 =>
+ classOf[(Any, Any, Any, Any) => Any].getName
+ case 5 =>
+ classOf[(Any, Any, Any, Any, Any) => Any].getName
+ case 6 =>
+ classOf[(Any, Any, Any, Any, Any, Any) => Any].getName
+ case 7 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 8 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 9 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 10 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 11 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 12 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 13 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 14 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 15 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 16 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 17 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 18 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 19 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 20 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 21 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case 22 =>
+ classOf[(Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any) => Any].getName
+ case _ =>
+ throw new UnsupportedOperationException(
+ "ScalaUDF doesn't support user-defined functions with more than 22 arguments")
+ }
+
+ val funcTerm = ctx.freshName("udf")
+ ctx.addMutableState(funcClassName, funcTerm,
+ s"this.$funcTerm = ($funcClassName)((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).userDefinedFunc());")
+
+ // codegen for children expressions
+ val evals = children.map(_.gen(ctx))
+
+ // Generate the codes for expressions and calling user-defined function
+ // We need to get the boxedType of dataType's javaType here. Because for the dataType
+ // such as IntegerType, its javaType is `int` and the returned type of user-defined
+ // function is Object. Trying to convert an Object to `int` will cause casting exception.
+ val evalCode = evals.map(_.code).mkString
+ val funcArguments = converterTerms.zip(evals).map {
+ case (converter, eval) => s"$converter.apply(${eval.value})"
+ }.mkString(",")
+ val callFunc = s"${ctx.boxedType(ctx.javaType(dataType))} $resultTerm = (${ctx.boxedType(ctx.javaType(dataType))})${catalystConverterTerm}.apply($funcTerm.apply($funcArguments));"
+
+ evalCode + s"""
+ ${ctx.javaType(dataType)} ${ev.value} = ${ctx.defaultValue(dataType)};
+ Boolean ${ev.isNull};
+
+ $callFunc
+
+ ${ev.value} = $resultTerm;
+ ${ev.isNull} = $resultTerm == null;
+ """
+ }
+
// scalastyle:on
--- End diff --
Could you move this line up (above the new code)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151059822
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44343/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154466223
@davies I think it would be better to do that in a follow up pr. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151034721
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153294018
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151023638
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151034722
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44339/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151025241
**[Test build #44337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44337/consoleFull)** for PR 9270 at commit [`b92ca97`](https://github.com/apache/spark/commit/b92ca97807220f7ac5b4b73f234b0e229f8fb3b1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153274594
**[Test build #44897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44897/consoleFull)** for PR 9270 at commit [`abb3750`](https://github.com/apache/spark/commit/abb375055b700c1cc431d093ed11ea536be44ce0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r43659185
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,122 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ // This must be called before children expressions' codegen
+ // because ctx.references is used in genCodeForConverter
+ val converterTerms = (0 until children.size).map(genCodeForConverter(ctx, _))
+
+ // Initialize user-defined function
+ val funcClassName = children.size match {
--- End diff --
This could be `scala.function${children.size}`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154249808
@viirya Do you have time to add more tests for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151059820
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151717301
@rxin any other comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r43658099
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,122 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
--- End diff --
Why `index` here? All freshName will be unique.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943415
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
Hmm, you are right. I should have it a try.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153103783
@viirya Could you add unit tests for every data types? it's easy to be wrong for some types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153276281
@viirya I meant the SQL data types, for example, DateType, ArrayType, StructType and so on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151031373
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151023782
**[Test build #44337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44337/consoleFull)** for PR 9270 at commit [`b92ca97`](https://github.com/apache/spark/commit/b92ca97807220f7ac5b4b73f234b0e229f8fb3b1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154499721
LGTM, merging this into master and 1.6 branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153294019
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44897/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154328642
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151023630
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153295860
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943201
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
Ah sorry. Can we just use scalaUDFClassName?
If not, we can create a static array in the beginning that covers all 22 versions of this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-150911128
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44315/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943173
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
But I think we don't have callFunc? I just created callFunc later which is the code used to invoke the function in codegen.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-152594395
ping @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153267637
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151040802
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-153270911
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-150899665
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-152802120
I'm traveling right now. Will take a look when I get back.
cc @davies also.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151025253
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9270#discussion_r42943239
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala ---
@@ -959,6 +963,861 @@ case class ScalaUDF(
}
}
+ // Generate codes used to convert the arguments to Scala type for user-defined funtions
+ private[this] def genCodeForConverter(ctx: CodeGenContext, index: Int): String = {
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+
+ val converterTerm = ctx.freshName("converter" + index.toString)
+ ctx.addMutableState(converterClassName, converterTerm,
+ s"this.$converterTerm = ($converterClassName)$typeConvertersClassName.createToScalaConverter(((${expressionClassName})((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).getChildren().apply($index))).dataType());")
+ converterTerm
+ }
+
+ override def genCode(
+ ctx: CodeGenContext,
+ ev: GeneratedExpressionCode): String = {
+
+ ctx.references += this
+
+ val scalaUDFClassName = classOf[ScalaUDF].getName
+ val converterClassName = classOf[Any => Any].getName
+ val typeConvertersClassName = CatalystTypeConverters.getClass.getName + ".MODULE$"
+ val expressionClassName = classOf[Expression].getName
+
+ // Generate codes used to convert the returned value of user-defined functions to Catalyst type
+ val catalystConverterTerm = ctx.freshName("catalystConverter")
+ ctx.addMutableState(converterClassName, catalystConverterTerm,
+ s"this.$catalystConverterTerm = ($converterClassName)$typeConvertersClassName.createToCatalystConverter((($scalaUDFClassName)expressions[${ctx.references.size - 1}]).dataType());")
+
+ val resultTerm = ctx.freshName("result")
+
+ val (evalCode, callFunc) = children.size match {
--- End diff --
Using scalaUDFClassName should not work because it is just `ScalaUDF`'s class name. I will try the static array approach later. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151029875
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-154328626
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9270#issuecomment-151025254
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44337/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org