You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by YanTangZhai <gi...@git.apache.org> on 2014/12/01 14:24:39 UTC
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
GitHub user YanTangZhai opened a pull request:
https://github.com/apache/spark/pull/3538
[SPARK-4676] [SQL] JavaSchemaRDD.schema may throw NullType MatchError if sql has null
val jsc = new org.apache.spark.api.java.JavaSparkContext(sc)
val jhc = new org.apache.spark.sql.hive.api.java.JavaHiveContext(jsc)
val nrdd = jhc.hql("select null from spark_test.for_test")
println(nrdd.schema)
Then the error is thrown as follows:
scala.MatchError: NullType (of class org.apache.spark.sql.catalyst.types.NullType$)
at org.apache.spark.sql.types.util.DataTypeConversions$.asJavaDataType(DataTypeConversions.scala:43)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/YanTangZhai/spark MatchNullType
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3538.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3538
----
commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-08-06T13:07:08Z
Merge pull request #1 from apache/master
update
commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-08-20T13:14:08Z
Merge pull request #3 from apache/master
Update
commit 8a0010691b669495b4c327cf83124cabb7da1405
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-09-12T06:54:58Z
Merge pull request #6 from apache/master
Update
commit 03b62b043ab7fd39300677df61c3d93bb9beb9e3
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-09-16T12:03:22Z
Merge pull request #7 from apache/master
Update
commit 76d40277d51f709247df1d3734093bf2c047737d
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-10-20T12:52:22Z
Merge pull request #8 from apache/master
update
commit d26d98248a1a4d0eb15336726b6f44e05dd7a05a
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-11-04T09:00:31Z
Merge pull request #9 from apache/master
Update
commit e249846d9b7967ae52ec3df0fb09e42ffd911a8a
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-11-11T03:18:24Z
Merge pull request #10 from apache/master
Update
commit 6e643f81555d75ec8ef3eb57bf5ecb6520485588
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-12-01T11:23:56Z
Merge pull request #11 from apache/master
Update
commit 896c7b73f0ba1b2d3dccf6fed6410bf077eb3d54
Author: yantangzhai <ty...@163.com>
Date: 2014-12-01T13:08:41Z
fix NullType MatchError in JavaSchemaRDD when sql has null
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3538#discussion_r21195689
--- Diff: sql/core/src/main/java/org/apache/spark/sql/api/java/DataType.java ---
@@ -82,6 +82,11 @@
*/
public static final ShortType ShortType = new ShortType();
+ /**
+ * Gets the NullType object.
+ */
+ public static final NullType NullType = new NullType();
--- End diff --
Indentation is off here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65072171
[Test build #23989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23989/consoleFull) for PR 3538 at commit [`896c7b7`](https://github.com/apache/spark/commit/896c7b73f0ba1b2d3dccf6fed6410bf077eb3d54).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65064263
[Test build #23989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23989/consoleFull) for PR 3538 at commit [`896c7b7`](https://github.com/apache/spark/commit/896c7b73f0ba1b2d3dccf6fed6410bf077eb3d54).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3538#discussion_r21122486
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/types/util/DataTypeConversions.scala ---
@@ -62,6 +62,7 @@ protected[sql] object DataTypeConversions {
case IntegerType => JDataType.IntegerType
case LongType => JDataType.LongType
case ShortType => JDataType.ShortType
+ case NullType => JDataType.StringType
--- End diff --
Why not also add a null type to JDataType?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65184515
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24021/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65184010
[Test build #24020 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24020/consoleFull) for PR 3538 at commit [`4b4bb34`](https://github.com/apache/spark/commit/4b4bb3400adb019dcd8d862bdd87fa5d3b06e805).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class NullType extends DataType `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65184511
[Test build #24021 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24021/consoleFull) for PR 3538 at commit [`e052dff`](https://github.com/apache/spark/commit/e052dff7df871bf6c8bc657ba3c9f424786cda8c).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class NullType extends DataType `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3538
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65180620
[Test build #24020 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24020/consoleFull) for PR 3538 at commit [`4b4bb34`](https://github.com/apache/spark/commit/4b4bb3400adb019dcd8d862bdd87fa5d3b06e805).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65072175
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23989/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65315929
Thanks for fixing this! I've merged to master and 1.2 (I fixed the indentation while merging).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65181173
[Test build #24021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24021/consoleFull) for PR 3538 at commit [`e052dff`](https://github.com/apache/spark/commit/e052dff7df871bf6c8bc657ba3c9f424786cda8c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65141433
It would also be good to add a test case for this to `JavaSQLSuite`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4676] [SQL] JavaSchemaRDD.schema may th...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3538#issuecomment-65184013
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24020/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org