You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by davies <gi...@git.apache.org> on 2014/10/28 07:26:37 UTC
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/2973
[SPARK-4133] [SQL] [PySpark] type conversionfor python udf
Call Python UDF on ArrayType/MapType/PrimitiveType, the returnType can also be ArrayType/MapType/PrimitiveType.
For StructType, it will act as tuple (without attributes). If returnType is StructType, it also should be tuple.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark udf_array
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2973.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2973
----
commit 35caa4fe7d87af752fb2df5bbd3e01e69784920d
Author: Davies Liu <da...@databricks.com>
Date: 2014-10-28T06:18:10Z
type conversionfor python udf
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60820309
Thanks for the quick fix! A few minor comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60836477
[Test build #22377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22377/consoleFull) for PR 2973 at commit [`11395fa`](https://github.com/apache/spark/commit/11395fa98549d49bc8c20ba2b01b33006449bc9b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60848314
[Test build #22377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22377/consoleFull) for PR 2973 at commit [`11395fa`](https://github.com/apache/spark/commit/11395fa98549d49bc8c20ba2b01b33006449bc9b).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class DateType(PrimitiveType):`
* `case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseNot(child: Expression) extends UnaryExpression `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60837916
[Test build #22379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22379/consoleFull) for PR 2973 at commit [`306956e`](https://github.com/apache/spark/commit/306956e0c11285f41b771bf4b2177345051248ae).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60720161
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22341/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60722293
[Test build #22352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22352/consoleFull) for PR 2973 at commit [`79afb4e`](https://github.com/apache/spark/commit/79afb4eb6bba998b95707b618d031af47e68bcc4).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60731528
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22352/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60849250
[Test build #489 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/489/consoleFull) for PR 2973 at commit [`2c00e43`](https://github.com/apache/spark/commit/2c00e43e2f1cbde7cd4d910ffba2d6871c514a1e).
* This patch **passes all tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `abstract class Broadcast[T: ClassTag](val id: Long) extends Serializable with Logging `
* `class VectorTransformer(object):`
* `class Normalizer(VectorTransformer):`
* `class JavaModelWrapper(VectorTransformer):`
* `class StandardScalerModel(JavaModelWrapper):`
* `class StandardScaler(object):`
* `class HashingTF(object):`
* `class IDFModel(JavaModelWrapper):`
* `class IDF(object):`
* `class Word2VecModel(JavaModelWrapper):`
* `class DateType(PrimitiveType):`
* `case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseNot(child: Expression) extends UnaryExpression `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2973#discussion_r19498538
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUdfs.scala ---
@@ -141,8 +213,11 @@ case class BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute], child:
val parent = childResults.mapPartitions { iter =>
val pickle = new Pickler
val currentRow = newMutableProjection(udf.children, child.output)()
+ val fields = udf.children.seq.map(_.dataType)
--- End diff --
Why `.seq`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60837255
[Test build #489 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/489/consoleFull) for PR 2973 at commit [`2c00e43`](https://github.com/apache/spark/commit/2c00e43e2f1cbde7cd4d910ffba2d6871c514a1e).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60865357
Thanks! Merged to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60848627
[Test build #22378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22378/consoleFull) for PR 2973 at commit [`2c00e43`](https://github.com/apache/spark/commit/2c00e43e2f1cbde7cd4d910ffba2d6871c514a1e).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class VectorTransformer(object):`
* `class Normalizer(VectorTransformer):`
* `class JavaModelWrapper(VectorTransformer):`
* `class StandardScalerModel(JavaModelWrapper):`
* `class StandardScaler(object):`
* `class HashingTF(object):`
* `class IDFModel(JavaModelWrapper):`
* `class IDF(object):`
* `class Word2VecModel(JavaModelWrapper):`
* `class DateType(PrimitiveType):`
* `case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseNot(child: Expression) extends UnaryExpression `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2973
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60731520
[Test build #22352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22352/consoleFull) for PR 2973 at commit [`79afb4e`](https://github.com/apache/spark/commit/79afb4eb6bba998b95707b618d031af47e68bcc4).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60721749
[Test build #485 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/485/consoleFull) for PR 2973 at commit [`35caa4f`](https://github.com/apache/spark/commit/35caa4fe7d87af752fb2df5bbd3e01e69784920d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60848633
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22378/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60849037
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22379/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60849033
[Test build #22379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22379/consoleFull) for PR 2973 at commit [`306956e`](https://github.com/apache/spark/commit/306956e0c11285f41b771bf4b2177345051248ae).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class DateType(PrimitiveType):`
* `case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseOr(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseXor(left: Expression, right: Expression) extends BinaryArithmetic `
* `case class BitwiseNot(child: Expression) extends UnaryExpression `
* `case class CreateTableAsSelect[T](`
* ` logDebug("Found class for $serdeName")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4113] [SQL] [PySpark] type conversionfo...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60875464
thanks, update the link
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60848321
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22377/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60714816
[Test build #22341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22341/consoleFull) for PR 2973 at commit [`35caa4f`](https://github.com/apache/spark/commit/35caa4fe7d87af752fb2df5bbd3e01e69784920d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60730509
[Test build #485 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/485/consoleFull) for PR 2973 at commit [`35caa4f`](https://github.com/apache/spark/commit/35caa4fe7d87af752fb2df5bbd3e01e69784920d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60865634
FYI this is actually: SPARK-4113
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2973#discussion_r19498469
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala ---
@@ -428,9 +397,10 @@ class SchemaRDD(
*/
private[sql] def collectToPython: JList[Array[Byte]] = {
val rowSchema = StructType.fromAttributes(this.queryExecution.analyzed.output)
+ val fields = rowSchema.fields.map(_.dataType)
--- End diff --
Nit/existing: I'd avoid creating attributes which is actually pretty expensive due to an `AtomicLong` and use the public API for `schema`. I think instead of these two lines you can just do the following here and above.
```scala
val fieldTypes = schema.fields.map(_.dataType)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60720156
[Test build #22341 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22341/consoleFull) for PR 2973 at commit [`35caa4f`](https://github.com/apache/spark/commit/35caa4fe7d87af752fb2df5bbd3e01e69784920d).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4133] [SQL] [PySpark] type conversionfo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2973#issuecomment-60837185
[Test build #22378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22378/consoleFull) for PR 2973 at commit [`2c00e43`](https://github.com/apache/spark/commit/2c00e43e2f1cbde7cd4d910ffba2d6871c514a1e).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org