You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by marmbrus <gi...@git.apache.org> on 2016/02/02 02:44:10 UTC
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/11014
[SPARK-13094][SQL] Add encoders for seq/array of primitives
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark seqEncoders
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11014.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11014
----
commit 1c2f67d06f5e70cf8277e2690c3205391059df31
Author: Michael Armbrust <mi...@databricks.com>
Date: 2016-02-02T01:42:34Z
[SPARK-13094][SQL] Add encoders for seq/array of primitives
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11014
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178360812
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178309506
**[Test build #50523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50523/consoleFull)** for PR 11014 at commit [`1c2f67d`](https://github.com/apache/spark/commit/1c2f67d06f5e70cf8277e2690c3205391059df31).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178374183
Can we add support for Java?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178386280
**[Test build #50538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50538/consoleFull)** for PR 11014 at commit [`0b5f03c`](https://github.com/apache/spark/commit/0b5f03c41ef7374c0a70771acb198ad61799fdec).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178377967
Sure, but I don't think we want to just enumerate in that API like we have to here for implicits to work. Instead we should do something like we do for tuples where you can compose. I'll defer that to [SPARK-13128](https://issues.apache.org/jira/browse/SPARK-13128) so this can make 1.6.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178360408
**[Test build #50528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50528/consoleFull)** for PR 11014 at commit [`ae356d1`](https://github.com/apache/spark/commit/ae356d103cd8264b449ad7b682c2af21fed6e2a4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178309734
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50523/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178412705
**[Test build #50538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50538/consoleFull)** for PR 11014 at commit [`0b5f03c`](https://github.com/apache/spark/commit/0b5f03c41ef7374c0a70771acb198ad61799fdec).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178360814
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50528/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178309726
**[Test build #50523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50523/consoleFull)** for PR 11014 at commit [`1c2f67d`](https://github.com/apache/spark/commit/1c2f67d06f5e70cf8277e2690c3205391059df31).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178413031
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50538/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11014#discussion_r51524739
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLImplicits.scala ---
@@ -56,13 +58,65 @@ abstract class SQLImplicits {
/** @since 1.6.0 */
implicit def newShortEncoder: Encoder[Short] = ExpressionEncoder()
- /** @since 1.6.0 */
+ /** @since 1.6.0 */
implicit def newBooleanEncoder: Encoder[Boolean] = ExpressionEncoder()
/** @since 1.6.0 */
implicit def newStringEncoder: Encoder[String] = ExpressionEncoder()
+ // Seqs
+
+ /** @since 1.6.1 */
+ implicit def newIntSeqEncoder: Encoder[Seq[Int]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newLongSeqEncoder: Encoder[Seq[Long]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newDoubleSeqEncoder: Encoder[Seq[Double]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newFloatSeqEncoder: Encoder[Seq[Float]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newByteSeqEncoder: Encoder[Seq[Byte]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newShortSeqEncoder: Encoder[Seq[Short]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newBooleanSeqEncoder: Encoder[Seq[Boolean]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newStringSeqEncoder: Encoder[Seq[String]] = ExpressionEncoder()
--- End diff --
Yeah, i went back and forth on that. I guess its probably useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178406498
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178309731
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11014#discussion_r51524706
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLImplicits.scala ---
@@ -56,13 +58,65 @@ abstract class SQLImplicits {
/** @since 1.6.0 */
implicit def newShortEncoder: Encoder[Short] = ExpressionEncoder()
- /** @since 1.6.0 */
+ /** @since 1.6.0 */
implicit def newBooleanEncoder: Encoder[Boolean] = ExpressionEncoder()
/** @since 1.6.0 */
implicit def newStringEncoder: Encoder[String] = ExpressionEncoder()
+ // Seqs
+
+ /** @since 1.6.1 */
+ implicit def newIntSeqEncoder: Encoder[Seq[Int]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newLongSeqEncoder: Encoder[Seq[Long]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newDoubleSeqEncoder: Encoder[Seq[Double]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newFloatSeqEncoder: Encoder[Seq[Float]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newByteSeqEncoder: Encoder[Seq[Byte]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newShortSeqEncoder: Encoder[Seq[Short]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newBooleanSeqEncoder: Encoder[Seq[Boolean]] = ExpressionEncoder()
+
+ /** @since 1.6.1 */
+ implicit def newStringSeqEncoder: Encoder[Seq[String]] = ExpressionEncoder()
--- End diff --
how about seq of product?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178413029
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13094][SQL] Add encoders for seq/array ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11014#issuecomment-178329483
**[Test build #50528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50528/consoleFull)** for PR 11014 at commit [`ae356d1`](https://github.com/apache/spark/commit/ae356d103cd8264b449ad7b682c2af21fed6e2a4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org