You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jackieMaKing <gi...@git.apache.org> on 2015/11/12 04:30:06 UTC
[GitHub] spark pull request: fixed SPARK-11679
GitHub user jackieMaKing opened a pull request:
https://github.com/apache/spark/pull/9649
fixed SPARK-11679
In the previous method, fields.toArray will cast java.util.List[StructField] into Array[Object] which can not cast into Array[StructField], thus when invoking this method will throw "java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Lorg.apache.spark.sql.types.StructField;"
I directly cast java.util.List[StructField] into Array[StructField] in this patch.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackieMaKing/spark Spark-11679
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9649.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9649
----
commit 1208354de8d4b07c1cc190f559965cf03e5e323f
Author: mayuanwen <ma...@qiyi.com>
Date: 2015-11-12T03:21:26Z
fixed SPARK-11679
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156317736
can you also add a test for it in `JavaDataFrameSuite`? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by jackieMaKing <gi...@git.apache.org>.
Github user jackieMaKing commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156308518
Call toArray(T[] a) in Java List to cast List to a specific type Array, thus I create a zero-length array. @srowen what do you mean, better to create an array with the same length of the list to pass to toArray? Like this "StructType(fields.toArray(new Array[StructField](fields.size())))"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9649#discussion_r44915401
--- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java ---
@@ -18,10 +18,7 @@
package test.org.apache.spark.sql;
import java.io.Serializable;
-import java.util.Arrays;
-import java.util.Comparator;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
--- End diff --
I generally wouldn't bother collapsing these imports. But don't change it unless you need to make other changes for this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156972147
**[Test build #45990 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45990/consoleFull)** for PR 9649 at commit [`4496753`](https://github.com/apache/spark/commit/4496753c491c67cdea2aa3b1f32567444354d593).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by jackieMaKing <gi...@git.apache.org>.
Github user jackieMaKing commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156671960
cloud-fun, I have add a test for SPARK-11679 in JavaDataFrameSuite.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156690562
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: fixed SPARK-11679
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-155986709
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9649#discussion_r44745705
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala ---
@@ -328,7 +328,8 @@ object StructType extends AbstractDataType {
def apply(fields: Seq[StructField]): StructType = StructType(fields.toArray)
def apply(fields: java.util.List[StructField]): StructType = {
- StructType(fields.toArray.asInstanceOf[Array[StructField]])
+ val temp = new Array[StructField](0)
+ StructType(fields.toArray(temp))
--- End diff --
I think we can `import scala.collection.JavaConverters._` and use `StructType(fields.asScala)` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: fixed SPARK-11679
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156083895
Please fix the title of this PR. Can you also not create a zero-length array each time? keep one instance; it's immutable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156690563
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45927/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156678981
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156680024
**[Test build #45927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45927/consoleFull)** for PR 9649 at commit [`c639ab4`](https://github.com/apache/spark/commit/c639ab47e8755cd8d2b2dd035541928233795350).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-157007674
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45990/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by jackieMaKing <gi...@git.apache.org>.
Github user jackieMaKing commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156312690
As suggested by cloud-fan, I pushed the new patch. I use scala.collection.JavaConverters to create StructType.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9649#discussion_r44860161
--- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java ---
@@ -209,6 +209,12 @@ public void testCreateDataFromFromList() {
Assert.assertEquals(1, result.length);
}
+ @Test
+ public void testCreateStructTypeFromList(){
+ StructType schema = StructType$.MODULE$.apply(Arrays.asList(createStructField("i", IntegerType, true)));
+ Assert.assertEquals(0, schema.fieldIndex("i"));
--- End diff --
I tried this test case locally, but can't reproduce the bug. However, the code snippet you provided in the JIRA do reproduce the bug. Can you investigate this and improve the test? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9649#discussion_r44918972
--- Diff: sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java ---
@@ -209,6 +206,18 @@ public void testCreateDataFromFromList() {
Assert.assertEquals(1, result.length);
}
+ @Test
+ public void testCreateStructTypeFromList(){
+ List<StructField> f_1 = new ArrayList<>();
+ f_1.add(new StructField("id", DataTypes.StringType, true, Metadata.empty()));
+ StructType schema1 = StructType$.MODULE$.apply(f_1);
+ Assert.assertEquals(0, schema1.fieldIndex("id"));
+
+ List<StructField> f_2 = Arrays.asList(new StructField("id", DataTypes.StringType, true, Metadata.empty()));
--- End diff --
how about `fields1` and `fields2`? The name `f_1` and `f_2` looks weird to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-157007672
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156690535
**[Test build #45927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45927/consoleFull)** for PR 9649 at commit [`c639ab4`](https://github.com/apache/spark/commit/c639ab47e8755cd8d2b2dd035541928233795350).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-157007505
**[Test build #45990 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45990/consoleFull)** for PR 9649 at commit [`4496753`](https://github.com/apache/spark/commit/4496753c491c67cdea2aa3b1f32567444354d593).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11679][SQL] Invoking method " apply(fie...
Posted by jackieMaKing <gi...@git.apache.org>.
Github user jackieMaKing commented on the pull request:
https://github.com/apache/spark/pull/9649#issuecomment-156973848
Oops！When create List<StructField> using Arrays.asList, we get a java.util.Arrays$ArrayList instead of java.util.ArrayList. The previous method works fine on java.util.Arrays$ArrayList but gets exception on java.util.ArrayList. This patch works fine with both.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org