You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by MaxGekk <gi...@git.apache.org> on 2018/04/27 15:55:00 UTC

[GitHub] spark pull request #21182: [SPARK-24068] Propagating DataFrameReader's optio...

GitHub user MaxGekk opened a pull request:

    https://github.com/apache/spark/pull/21182

    [SPARK-24068] Propagating DataFrameReader's options to Text datasource on schema inferring

    ## What changes were proposed in this pull request?
    
    While reading CSV or JSON files, DataFrameReader's options are converted to Hadoop's parameters, for example there:
    https://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L302
    
    but the options are not propagated to Text datasource on schema inferring, for instance:
    https://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala#L184-L188
    
    The PR proposes propagation of user's options to Text datasource on scheme inferring in similar way as user's options are converted to Hadoop parameters if schema is specified.
    
    ## How was this patch tested?
    The changes were tested manually by using https://github.com/twitter/hadoop-lzo:
    
    ```
    hadoop-lzo> mvn clean package
    hadoop-lzo> ln -s ./target/hadoop-lzo-0.4.21-SNAPSHOT.jar ./hadoop-lzo.jar
    ```
    Create 2 test files in JSON and CSV format and compress them:
    ```shell
    $ cat test.csv
    col1|col2
    a|1
    $ lzop test.csv
    $ cat test.json
    {"col1":"a","col2":1}
    $ lzop test.json
    ```
    Run `spark-shell` with hadoop-lzo:
    ```
    bin/spark-shell --jars ~/hadoop-lzo/hadoop-lzo.jar
    ```
    reading compressing CSV and JSON without schema:
    ```scala
    spark.read.option("io.compression.codecs", "com.hadoop.compression.lzo.LzopCodec").option("inferSchema",true).option("header",true).option("sep","|").csv("test.csv.lzo").show()
    +----+----+
    |col1|col2|
    +----+----+
    |   a|   1|
    +----+----+
    ```
    ```scala
    spark.read.option("io.compression.codecs", "com.hadoop.compression.lzo.LzopCodec").option("multiLine", true).json("test.json.lzo").printSchema
    root
     |-- col1: string (nullable = true)
     |-- col2: long (nullable = true)
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/MaxGekk/spark-1 text-options

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21182.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21182
    
----
commit 8a8ff3f5bfdfaee7ec73e362cfa34261d199f407
Author: Maxim Gekk <ma...@...>
Date:   2018-04-27T13:23:40Z

    Propagating DataFrameReader's options to the text datasource on schema inferring

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #21182: [SPARK-24068] Propagating DataFrameReader's optio...

Posted by viirya <gi...@git.apache.org>.

Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21182#discussion_r186263531
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala ---
    @@ -136,4 +136,6 @@ private[sql] class JSONOptions(
           allowBackslashEscapingAnyCharacter)
         factory.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, allowUnquotedControlChars)
       }
    +
    +  @transient val textOptions = parameters
    --- End diff --
    
    If you want to access `parameters` like this, why not just remove `private` on it?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21182: [SPARK-24068] Propagating DataFrameReader's options to T...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21182
  
    **[Test build #90010 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90010/testReport)** for PR 21182 at commit [`9f55aa8`](https://github.com/apache/spark/commit/9f55aa8fa9589bd23fddfe83f9badff02492e69c).
     * This patch **fails SparkR unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21182: [SPARK-24068] Propagating DataFrameReader's options to T...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21182
  
    **[Test build #89953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89953/testReport)** for PR 21182 at commit [`1fa871d`](https://github.com/apache/spark/commit/1fa871dd4cad9a04cf8617031ab47844a82bb56e).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21182: [SPARK-24068] Propagating DataFrameReader's options to T...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21182
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89953/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21182: [SPARK-24068] Propagating DataFrameReader's options to T...

Posted by MaxGekk <gi...@git.apache.org>.

Github user MaxGekk commented on the issue:

    https://github.com/apache/spark/pull/21182
  
    Here is the backport to 2.3: https://github.com/apache/spark/pull/21292


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21182: [SPARK-24068] Propagating DataFrameReader's options to T...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21182
  
    **[Test build #89942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89942/testReport)** for PR 21182 at commit [`8a8ff3f`](https://github.com/apache/spark/commit/8a8ff3f5bfdfaee7ec73e362cfa34261d199f407).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21182: [SPARK-24068] Propagating DataFrameReader's options to T...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21182
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90284/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21182: [SPARK-24068] Propagating DataFrameReader's options to T...

Posted by MaxGekk <gi...@git.apache.org>.

Github user MaxGekk commented on the issue:

    https://github.com/apache/spark/pull/21182
  
    jenkins, retest this, please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21182: [SPARK-24068] Propagating DataFrameReader's options to T...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21182
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org