You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by MaxGekk <gi...@git.apache.org> on 2018/05/20 17:14:37 UTC
[GitHub] spark pull request #21377: [SPARK-24325] Tests for Hadoop's LineReader
GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/21377
[SPARK-24325] Tests for Hadoop's LineReader
## What changes were proposed in this pull request?
The tests cover basic functionality of [Hadoop LinesReader](https://github.com/apache/spark/blob/8d79113b812a91073d2c24a3a9ad94cc3b90b24a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFileLinesReader.scala#L42). In particular, the added tests check:
- A split slices a line or delimiter
- A split slices two consecutive lines and cover a delimiter between the lines
- Two splits slice a line and there are no duplicates
- Internal buffer size (`io.file.buffer.size`) is less than line length
- Constrain of maximum line length - `mapreduce.input.linerecordreader.line.maxlength`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 line-reader-tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21377.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21377
----
commit 31ac9ca4e992f234df63d09c2934919f24fe20d4
Author: Maxim Gekk <ma...@...>
Date: 2018-05-20T16:55:04Z
Tests for Hadoop's LineReader
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21377
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90867/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21377
**[Test build #90867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90867/testReport)** for PR 21377 at commit [`31ac9ca`](https://github.com/apache/spark/commit/31ac9ca4e992f234df63d09c2934919f24fe20d4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HadoopFileLinesReaderSuite extends SharedSQLContext `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/21377
jenkins, retest this, please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21377
**[Test build #90867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90867/testReport)** for PR 21377 at commit [`31ac9ca`](https://github.com/apache/spark/commit/31ac9ca4e992f234df63d09c2934919f24fe20d4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LineReader
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21377
**[Test build #90865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90865/testReport)** for PR 21377 at commit [`31ac9ca`](https://github.com/apache/spark/commit/31ac9ca4e992f234df63d09c2934919f24fe20d4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21377
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LineReader
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21377
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21377
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21377
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21377
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90865/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21377: [SPARK-24325] Tests for Hadoop's LinesReader
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21377
**[Test build #90865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90865/testReport)** for PR 21377 at commit [`31ac9ca`](https://github.com/apache/spark/commit/31ac9ca4e992f234df63d09c2934919f24fe20d4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HadoopFileLinesReaderSuite extends SharedSQLContext `
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org