You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by MaxGekk <gi...@git.apache.org> on 2018/05/22 12:11:48 UTC

[GitHub] spark pull request #21394: [SPARK-24329][SQL] Test for skipping multi-space ...

GitHub user MaxGekk opened a pull request:

    https://github.com/apache/spark/pull/21394

    [SPARK-24329][SQL] Test for skipping multi-space lines

    ## What changes were proposed in this pull request?
    
    The PR is a continue of https://github.com/apache/spark/pull/21380 . It checks cases that are handled by the code:
    https://github.com/apache/spark/blob/e3de6ab30d52890eb08578e55eb4a5d2b4e7aa35/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala#L303-L304
    
    Basically the code skips lines with one or many whitespaces, and lines with comments (see [filterCommentAndEmpty](https://github.com/apache/spark/blob/e3de6ab30d52890eb08578e55eb4a5d2b4e7aa35/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVUtils.scala#L47))
    
    ```scala
       iter.filter { line =>
          line.trim.nonEmpty && !line.startsWith(options.comment.toString)
        }
    ```
    ## How was this patch tested?
    
    Added a test for the case described above.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/MaxGekk/spark-1 test-for-multi-space-lines

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21394.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21394
    
----
commit b0f73e5f5dda5ec74c91dad07f50f9960402cc82
Author: Maxim Gekk <ma...@...>
Date:   2018-05-22T11:59:51Z

    Test checks skipping lines with comments, and one or multiple whitespaces

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    **[Test build #90954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90954/testReport)** for PR 21394 at commit [`b0f73e5`](https://github.com/apache/spark/commit/b0f73e5f5dda5ec74c91dad07f50f9960402cc82).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    **[Test build #91032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91032/testReport)** for PR 21394 at commit [`470980f`](https://github.com/apache/spark/commit/470980fbcea007de290af74acd7503080d2ee0c3).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    **[Test build #91063 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91063/testReport)** for PR 21394 at commit [`a2cdc29`](https://github.com/apache/spark/commit/a2cdc2916d70c7031f41b1b9795af0e4b50cc977).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `final class PrefixSpan(@Since(\"2.4.0\") override val uid: String) extends Params `
      * `abstract class CodeGeneratorWithInterpretedFallback[IN, OUT] `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90954/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    **[Test build #91032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91032/testReport)** for PR 21394 at commit [`470980f`](https://github.com/apache/spark/commit/470980fbcea007de290af74acd7503080d2ee0c3).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    **[Test build #91050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91050/testReport)** for PR 21394 at commit [`470980f`](https://github.com/apache/spark/commit/470980fbcea007de290af74acd7503080d2ee0c3).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90997/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    **[Test build #90988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90988/testReport)** for PR 21394 at commit [`70839e8`](https://github.com/apache/spark/commit/70839e8bf56338698d750a9d7830dca29ae13dce).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `case class TaskKilled(`
      * `trait HasValidationIndicatorCol extends Params `
      * `trait DivModLike extends BinaryArithmetic `
      * `case class Divide(left: Expression, right: Expression) extends DivModLike `
      * `case class Remainder(left: Expression, right: Expression) extends DivModLike `
      * `class ReadOnlySQLConf(context: TaskContext) extends SQLConf `
      * `class TaskContextConfigProvider(context: TaskContext) extends ConfigProvider `
      * `case class ContinuousShuffleReadPartition(index: Int, queueSize: Int) extends Partition `
      * `class ContinuousShuffleReadRDD(`
      * `trait ContinuousShuffleReader `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    **[Test build #90971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90971/testReport)** for PR 21394 at commit [`70839e8`](https://github.com/apache/spark/commit/70839e8bf56338698d750a9d7830dca29ae13dce).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `case class TaskKilled(`
      * `trait HasValidationIndicatorCol extends Params `
      * `trait DivModLike extends BinaryArithmetic `
      * `case class Divide(left: Expression, right: Expression) extends DivModLike `
      * `case class Remainder(left: Expression, right: Expression) extends DivModLike `
      * `class ReadOnlySQLConf(context: TaskContext) extends SQLConf `
      * `class TaskContextConfigProvider(context: TaskContext) extends ConfigProvider `
      * `case class ContinuousShuffleReadPartition(index: Int, queueSize: Int) extends Partition `
      * `class ContinuousShuffleReadRDD(`
      * `trait ContinuousShuffleReader `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by kiszk <gi...@git.apache.org>.

Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    Retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21394: [SPARK-24329][SQL] Test for skipping multi-space lines

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/21394
  
    Sounds reasonable for now. LGTM.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org