You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2017/11/13 02:52:51 UTC

[GitHub] spark pull request #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection sh...

GitHub user viirya opened a pull request:

    https://github.com/apache/spark/pull/19734

    [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should produce correct field names for special characters

    ## What changes were proposed in this pull request?
    
    For a class with field name of special characters, e.g.:
    ```scala
    case class MyType(`field.1`: String, `field 2`: String)
    ```
    
    Although we can manipulate DataFrame/Dataset, the field names are encoded:
    ```scala
    scala> val df = Seq(MyType("a", "b"), MyType("c", "d")).toDF
    df: org.apache.spark.sql.DataFrame = [field$u002E1: string, field$u00202: string]
    scala> df.as[MyType].collect
    res7: Array[MyType] = Array(MyType(a,b), MyType(c,d))
    ```
    
    It causes resolving problem when we try to convert the data with non-encoded field names:
    ```scala
    spark.read.json(path).as[MyType]
    ...
    [info]   org.apache.spark.sql.AnalysisException: cannot resolve '`field$u002E1`' given input columns: [field 2, fie
    ld.1];
    [info]   at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
    ...
    ```
    
    We should use decoded field name in Dataset schema.
    
    ## How was this patch tested?
    
    Added tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/viirya/spark-1 SPARK-22442-2.2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19734.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19734
    
----
commit 8d3fd950ca76d791335b9000133d0b1f897d2f87
Author: Liang-Chi Hsieh <vi...@gmail.com>
Date:   2017-11-13T02:40:01Z

    ScalaReflection should produce correct field names for special characters.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should pr...

Posted by viirya <gi...@git.apache.org>.

Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/19734
  
    cc @cloud-fan The backport of SPARK-22442 to branch 2.2.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should pr...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19734
  
    **[Test build #83763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83763/testReport)** for PR 19734 at commit [`8d3fd95`](https://github.com/apache/spark/commit/8d3fd950ca76d791335b9000133d0b1f897d2f87).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should pr...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on the issue:

    https://github.com/apache/spark/pull/19734
  
    ok thanks, I'm above to tag 2.2.1. technically this isn't a regression but I could wait for a few hours (need to wait for jenkins build from the branch) if we could merge this ASAP


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should pr...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19734
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83763/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should pr...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19734
  
    **[Test build #83763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83763/testReport)** for PR 19734 at commit [`8d3fd95`](https://github.com/apache/spark/commit/8d3fd950ca76d791335b9000133d0b1f897d2f87).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should pr...

Posted by viirya <gi...@git.apache.org>.

Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/19734
  
    @felixcheung Yes. Looking into it.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should pr...

Posted by viirya <gi...@git.apache.org>.

Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/19734
  
    Thanks @felixcheung 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #19734: [SPARK-22442][SQL][BRANCH-2.2] ScalaReflection should pr...

Posted by felixcheung <gi...@git.apache.org>.

Github user felixcheung commented on the issue:

    https://github.com/apache/spark/pull/19734
  
    @viirya could you take a look https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.2-compile-sbt-scala-2.10/724/



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org