You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by MaxGekk <gi...@git.apache.org> on 2018/04/12 17:41:03 UTC
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/21056
[SPARK-23849][SQL] Tests for samplingRatio of json datasource
## What changes were proposed in this pull request?
Added the `samplingRatio` option to the `json()` method of PySpark DataFrame Reader. Improving existing tests for Scala API according to review of the PR: https://github.com/apache/spark/pull/20959
## How was this patch tested?
Added new test for PySpark, updated 2 existing tests according to reviews of https://github.com/apache/spark/pull/20959 and added new negative test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 json-sampling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21056.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21056
----
commit 95dcdee31e62ff3d01beef20681d875dede8537d
Author: Maxim Gekk <ma...@...>
Date: 2018-04-12T16:47:44Z
Making changes according to Hyukjin Kwon's review
commit a31b2e20c049d8c4d10d1a5880f0ec1d631ceea0
Author: Maxim Gekk <ma...@...>
Date: 2018-04-12T17:26:04Z
Adding samplingRatio as an option for the json method
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21056
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89674/testReport)** for PR 21056 at commit [`f96134c`](https://github.com/apache/spark/commit/f96134c39adf643148c87f9bf7f0d5340b0219a3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89285/testReport)** for PR 21056 at commit [`a31b2e2`](https://github.com/apache/spark/commit/a31b2e20c049d8c4d10d1a5880f0ec1d631ceea0).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89366/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89366/testReport)** for PR 21056 at commit [`5c1d2b2`](https://github.com/apache/spark/commit/5c1d2b2179a1609b17c8e47b2abeef8a298c1643).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r184010158
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,67 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val ds = sampledTestData.coalesce(1)
+ ds.write.text(path.getAbsolutePath)
+ val readback = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ })
+ }
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
- assert(ds.schema == new StructType().add("f1", LongType))
- }
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val ds = sampledTestData.coalesce(1)
+ val readback = spark.read.option("samplingRatio", 0.1).json(ds)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val ds = spark.range(0, 100, 1, 1).map(_.toString)
+
+ val errorMsg0 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", -1).json(ds)
+ }.getMessage
+ assert(errorMsg0.contains("samplingRatio (-1.0) should be greater than 0"))
- assert(ds.schema == new StructType().add("f1", LongType))
+ val errorMsg1 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", 0).json(ds)
+ }.getMessage
+ assert(errorMsg1.contains("samplingRatio (0.0) should be greater than 0"))
+
+ val sampled = spark.read.option("samplingRatio", 10).json(ds)
+ assert(sampled.count() == ds.count())
+ }
+
+ test("SPARK-23849: sampling files for schema inferring in the multiLine mode") {
+ withTempDir { dir =>
+ Files.write(Paths.get(dir.getAbsolutePath, "0.json"),
+ """{"a":"a"}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ for (i <- 1 until 10) {
+ Files.write(Paths.get(dir.getAbsolutePath, s"$i.json"),
+ s"""{"a":$i}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ }
+ val files = (0 until 10).map { i =>
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ val hdfsPath = new Path(Paths.get(dir.getAbsolutePath, s"$i.json").toString)
+ hdfsPath.getFileSystem(hadoopConf).getFileStatus(hdfsPath)
+ }
+ // The test uses the internal method because public API cannot guarantee order of files
+ // passed to the infer method. The order is changed between runs because the temporary
+ // folder has different path which leads to different order of file statuses returned
--- End diff --
> Ah, do you mean it's not guaranteed if we get the file statues in such order or not, via, for example Hadoop API?
Yes, I am not sure we can guarantee that but frankly speaking I am not sure that sequence of files can guarantee that too. ;-)
> I took a look about this before and found it's designed to be in an alphabetical order although we shouldn't rely on this order if I remember this correctly.
Probably you are right. I ran the test 100 times by passing dir successfully but it says nothing. In another environment it can be flaky.
> Even if so, then we can't gurantee `RDD[PortableDataStream]` is a single partition within `MultiLineJsonDataSource.infer`.
Right. From another hand it is pretty bad that we cannot control number of partitions and its sizes. Let's image sampled input is big enough and isn't evenly distributed across partitions. We will have the situation when most part of schema inferring job is performed by one task. Maybe it is better to repartition sampled RDD/Dataset before doing schema inferring. What do you think of it?
> it's okay if it's difficult to write a test. Manual test and updating PR description is a-okay.
Would you propose to delete the test?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r184017470
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,67 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val ds = sampledTestData.coalesce(1)
+ ds.write.text(path.getAbsolutePath)
+ val readback = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ })
+ }
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
- assert(ds.schema == new StructType().add("f1", LongType))
- }
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val ds = sampledTestData.coalesce(1)
+ val readback = spark.read.option("samplingRatio", 0.1).json(ds)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val ds = spark.range(0, 100, 1, 1).map(_.toString)
+
+ val errorMsg0 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", -1).json(ds)
+ }.getMessage
+ assert(errorMsg0.contains("samplingRatio (-1.0) should be greater than 0"))
- assert(ds.schema == new StructType().add("f1", LongType))
+ val errorMsg1 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", 0).json(ds)
+ }.getMessage
+ assert(errorMsg1.contains("samplingRatio (0.0) should be greater than 0"))
+
+ val sampled = spark.read.option("samplingRatio", 10).json(ds)
+ assert(sampled.count() == ds.count())
+ }
+
+ test("SPARK-23849: sampling files for schema inferring in the multiLine mode") {
+ withTempDir { dir =>
+ Files.write(Paths.get(dir.getAbsolutePath, "0.json"),
+ """{"a":"a"}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ for (i <- 1 until 10) {
+ Files.write(Paths.get(dir.getAbsolutePath, s"$i.json"),
+ s"""{"a":$i}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ }
+ val files = (0 until 10).map { i =>
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ val hdfsPath = new Path(Paths.get(dir.getAbsolutePath, s"$i.json").toString)
+ hdfsPath.getFileSystem(hadoopConf).getFileStatus(hdfsPath)
+ }
+ // The test uses the internal method because public API cannot guarantee order of files
+ // passed to the infer method. The order is changed between runs because the temporary
+ // folder has different path which leads to different order of file statuses returned
--- End diff --
Yup. I think it's fine to delete.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89674/testReport)** for PR 21056 at commit [`f96134c`](https://github.com/apache/spark/commit/f96134c39adf643148c87f9bf7f0d5340b0219a3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89289/testReport)** for PR 21056 at commit [`6aa36d4`](https://github.com/apache/spark/commit/6aa36d457a7bec5cec3c974ba316f267fb04157e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89366/testReport)** for PR 21056 at commit [`5c1d2b2`](https://github.com/apache/spark/commit/5c1d2b2179a1609b17c8e47b2abeef8a298c1643).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89692/testReport)** for PR 21056 at commit [`1b86df3`](https://github.com/apache/spark/commit/1b86df3293612ef1db80220c8d8e71a4b917a5c7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r183982653
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,67 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val ds = sampledTestData.coalesce(1)
+ ds.write.text(path.getAbsolutePath)
+ val readback = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ })
+ }
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
- assert(ds.schema == new StructType().add("f1", LongType))
- }
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val ds = sampledTestData.coalesce(1)
+ val readback = spark.read.option("samplingRatio", 0.1).json(ds)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val ds = spark.range(0, 100, 1, 1).map(_.toString)
+
+ val errorMsg0 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", -1).json(ds)
+ }.getMessage
+ assert(errorMsg0.contains("samplingRatio (-1.0) should be greater than 0"))
- assert(ds.schema == new StructType().add("f1", LongType))
+ val errorMsg1 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", 0).json(ds)
+ }.getMessage
+ assert(errorMsg1.contains("samplingRatio (0.0) should be greater than 0"))
+
+ val sampled = spark.read.option("samplingRatio", 10).json(ds)
+ assert(sampled.count() == ds.count())
+ }
+
+ test("SPARK-23849: sampling files for schema inferring in the multiLine mode") {
+ withTempDir { dir =>
+ Files.write(Paths.get(dir.getAbsolutePath, "0.json"),
+ """{"a":"a"}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ for (i <- 1 until 10) {
+ Files.write(Paths.get(dir.getAbsolutePath, s"$i.json"),
+ s"""{"a":$i}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ }
+ val files = (0 until 10).map { i =>
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ val hdfsPath = new Path(Paths.get(dir.getAbsolutePath, s"$i.json").toString)
+ hdfsPath.getFileSystem(hadoopConf).getFileStatus(hdfsPath)
+ }
+ // The test uses the internal method because public API cannot guarantee order of files
+ // passed to the infer method. The order is changed between runs because the temporary
+ // folder has different path which leads to different order of file statuses returned
--- End diff --
> the temporary folder has different path which leads to different order of file statuses
Can we just read `dir` since we explicitly wrote the files above?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r183227071
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,77 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
+ val sampledTestData = (value: java.lang.Long) => {
--- End diff --
@MaxGekk, can we have the data in `TestJsonData`, for example,
```scala
def sampledTestData: Dataset[String] =
spark.createDataset(spark.sparkContext.parallelize(
...
)(Encoders.STRING)
```
and use it, for example, `sampledTestData.coalesce(1)` in `JsonSuite`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89298/testReport)** for PR 21056 at commit [`6aa36d4`](https://github.com/apache/spark/commit/6aa36d457a7bec5cec3c974ba316f267fb04157e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89289/testReport)** for PR 21056 at commit [`6aa36d4`](https://github.com/apache/spark/commit/6aa36d457a7bec5cec3c974ba316f267fb04157e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89861/testReport)** for PR 21056 at commit [`2bd14e2`](https://github.com/apache/spark/commit/2bd14e26adbf8d7e306b1f8915e059c3e8f01fb3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89692/testReport)** for PR 21056 at commit [`1b86df3`](https://github.com/apache/spark/commit/1b86df3293612ef1db80220c8d8e71a4b917a5c7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21056
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r183998567
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,67 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val ds = sampledTestData.coalesce(1)
+ ds.write.text(path.getAbsolutePath)
+ val readback = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ })
+ }
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
- assert(ds.schema == new StructType().add("f1", LongType))
- }
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val ds = sampledTestData.coalesce(1)
+ val readback = spark.read.option("samplingRatio", 0.1).json(ds)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val ds = spark.range(0, 100, 1, 1).map(_.toString)
+
+ val errorMsg0 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", -1).json(ds)
+ }.getMessage
+ assert(errorMsg0.contains("samplingRatio (-1.0) should be greater than 0"))
- assert(ds.schema == new StructType().add("f1", LongType))
+ val errorMsg1 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", 0).json(ds)
+ }.getMessage
+ assert(errorMsg1.contains("samplingRatio (0.0) should be greater than 0"))
+
+ val sampled = spark.read.option("samplingRatio", 10).json(ds)
+ assert(sampled.count() == ds.count())
+ }
+
+ test("SPARK-23849: sampling files for schema inferring in the multiLine mode") {
+ withTempDir { dir =>
+ Files.write(Paths.get(dir.getAbsolutePath, "0.json"),
+ """{"a":"a"}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ for (i <- 1 until 10) {
+ Files.write(Paths.get(dir.getAbsolutePath, s"$i.json"),
+ s"""{"a":$i}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ }
+ val files = (0 until 10).map { i =>
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ val hdfsPath = new Path(Paths.get(dir.getAbsolutePath, s"$i.json").toString)
+ hdfsPath.getFileSystem(hadoopConf).getFileStatus(hdfsPath)
+ }
+ // The test uses the internal method because public API cannot guarantee order of files
+ // passed to the infer method. The order is changed between runs because the temporary
+ // folder has different path which leads to different order of file statuses returned
--- End diff --
Ah, do you mean it's not guaranteed if we get the file statues in such order or not? Even if so, then we can gurantee `RDD[PortableDataStream]` is a single partition within `MultiLineJsonDataSource.infer` .. @MaxGekk, it's okay if it's difficult to write a test. Manual test and updating PR description is a-okay.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89678/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r184019041
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,67 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val ds = sampledTestData.coalesce(1)
+ ds.write.text(path.getAbsolutePath)
+ val readback = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ })
+ }
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
- assert(ds.schema == new StructType().add("f1", LongType))
- }
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val ds = sampledTestData.coalesce(1)
+ val readback = spark.read.option("samplingRatio", 0.1).json(ds)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val ds = spark.range(0, 100, 1, 1).map(_.toString)
+
+ val errorMsg0 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", -1).json(ds)
+ }.getMessage
+ assert(errorMsg0.contains("samplingRatio (-1.0) should be greater than 0"))
- assert(ds.schema == new StructType().add("f1", LongType))
+ val errorMsg1 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", 0).json(ds)
+ }.getMessage
+ assert(errorMsg1.contains("samplingRatio (0.0) should be greater than 0"))
+
+ val sampled = spark.read.option("samplingRatio", 10).json(ds)
+ assert(sampled.count() == ds.count())
+ }
+
+ test("SPARK-23849: sampling files for schema inferring in the multiLine mode") {
+ withTempDir { dir =>
+ Files.write(Paths.get(dir.getAbsolutePath, "0.json"),
+ """{"a":"a"}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ for (i <- 1 until 10) {
+ Files.write(Paths.get(dir.getAbsolutePath, s"$i.json"),
+ s"""{"a":$i}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ }
+ val files = (0 until 10).map { i =>
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ val hdfsPath = new Path(Paths.get(dir.getAbsolutePath, s"$i.json").toString)
+ hdfsPath.getFileSystem(hadoopConf).getFileStatus(hdfsPath)
+ }
+ // The test uses the internal method because public API cannot guarantee order of files
+ // passed to the infer method. The order is changed between runs because the temporary
+ // folder has different path which leads to different order of file statuses returned
--- End diff --
for repartition in schena inference, I dont feel strongly. lets monitor JIRAs and mailing list and see if there are some requests for it. it sounds too detailed control to me for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89285/testReport)** for PR 21056 at commit [`a31b2e2`](https://github.com/apache/spark/commit/a31b2e20c049d8c4d10d1a5880f0ec1d631ceea0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89861/testReport)** for PR 21056 at commit [`2bd14e2`](https://github.com/apache/spark/commit/2bd14e26adbf8d7e306b1f8915e059c3e8f01fb3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89298/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/21056
jenkins, retest this, please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r183227100
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,77 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
+ val sampledTestData = (value: java.lang.Long) => {
+ val predefinedSample = Set[Long](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
-
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
- assert(ds.schema == new StructType().add("f1", LongType))
+ if (predefinedSample.contains(value)) {
+ s"""{"f1":${value.toString}}"""
+ } else {
+ s"""{"f1":${(value.toDouble + 0.1).toString}}"""
}
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val ds = spark.range(0, 100, 1, 1).map(sampledTestData)
+ ds.write.text(path.getAbsolutePath)
+ val readback = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ })
+ }
+
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val ds = spark.range(0, 100, 1, 1).map(sampledTestData)
+ val readback = spark.read.option("samplingRatio", 0.1).json(ds)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ }
+
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val ds = spark.range(0, 100, 1, 1).map(_.toString)
- assert(ds.schema == new StructType().add("f1", LongType))
+ val errorMsg0 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", -1).json(ds)
+ }.getMessage
+ assert(errorMsg0.contains("samplingRatio (-1.0) should be greater than 0"))
+
+ val errorMsg1 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", 0).json(ds)
+ }.getMessage
+ assert(errorMsg1.contains("samplingRatio (0.0) should be greater than 0"))
+
+ val sampled = spark.read.option("samplingRatio", 10).json(ds)
+ assert(sampled.count() == ds.count())
+ }
+
+ test("SPARK-23849: sampling files for schema inferring in the multiLine mode") {
+ withTempDir { dir =>
+ Files.write(Paths.get(dir.getAbsolutePath, "0.json"), """{"a":"a"}""".getBytes,
--- End diff --
maybe `getBytes` with explicit encoding UTF-8.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89678/testReport)** for PR 21056 at commit [`fdeac84`](https://github.com/apache/spark/commit/fdeac84f5b6fe2e25b32cbed4d1771e7c85887cc).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89674/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89289/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89678/testReport)** for PR 21056 at commit [`fdeac84`](https://github.com/apache/spark/commit/fdeac84f5b6fe2e25b32cbed4d1771e7c85887cc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r184002553
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -239,6 +239,9 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None,
including tab and line feed characters) or not.
:param lineSep: defines the line separator that should be used for parsing. If None is
set, it covers all ``\\r``, ``\\r\\n`` and ``\\n``.
+ :param samplingRatio: defines fraction of rows (when ``multiLine`` is ``false``) or fraction
--- End diff --
Will it be ok if I write something like:
```
samplingRation defines fraction of input json objects used for schema inferring
```
Please, give me your variants if it doesn't work for you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r183994877
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,67 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val ds = sampledTestData.coalesce(1)
+ ds.write.text(path.getAbsolutePath)
+ val readback = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ })
+ }
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
- assert(ds.schema == new StructType().add("f1", LongType))
- }
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val ds = sampledTestData.coalesce(1)
+ val readback = spark.read.option("samplingRatio", 0.1).json(ds)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val ds = spark.range(0, 100, 1, 1).map(_.toString)
+
+ val errorMsg0 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", -1).json(ds)
+ }.getMessage
+ assert(errorMsg0.contains("samplingRatio (-1.0) should be greater than 0"))
- assert(ds.schema == new StructType().add("f1", LongType))
+ val errorMsg1 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", 0).json(ds)
+ }.getMessage
+ assert(errorMsg1.contains("samplingRatio (0.0) should be greater than 0"))
+
+ val sampled = spark.read.option("samplingRatio", 10).json(ds)
+ assert(sampled.count() == ds.count())
+ }
+
+ test("SPARK-23849: sampling files for schema inferring in the multiLine mode") {
+ withTempDir { dir =>
+ Files.write(Paths.get(dir.getAbsolutePath, "0.json"),
+ """{"a":"a"}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ for (i <- 1 until 10) {
+ Files.write(Paths.get(dir.getAbsolutePath, s"$i.json"),
+ s"""{"a":$i}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ }
+ val files = (0 until 10).map { i =>
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ val hdfsPath = new Path(Paths.get(dir.getAbsolutePath, s"$i.json").toString)
+ hdfsPath.getFileSystem(hadoopConf).getFileStatus(hdfsPath)
+ }
+ // The test uses the internal method because public API cannot guarantee order of files
+ // passed to the infer method. The order is changed between runs because the temporary
+ // folder has different path which leads to different order of file statuses returned
--- End diff --
Maybe I missed something but if the order by different paths were problem, that's fixed by explicit file name above. Then, I thought we could just use `spark.read`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21056
**[Test build #89298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89298/testReport)** for PR 21056 at commit [`6aa36d4`](https://github.com/apache/spark/commit/6aa36d457a7bec5cec3c974ba316f267fb04157e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r183977583
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -239,6 +239,9 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None,
including tab and line feed characters) or not.
:param lineSep: defines the line separator that should be used for parsing. If None is
set, it covers all ``\\r``, ``\\r\\n`` and ``\\n``.
+ :param samplingRatio: defines fraction of rows (when ``multiLine`` is ``false``) or fraction
--- End diff --
@MaxGekk, can we just don't say it's for files when `multiLine` is enabled? I think JSON makes a sample for each record (or an array of JSONs) regardless of record per file or multiple records in a file at high level.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r184018038
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -239,6 +239,9 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None,
including tab and line feed characters) or not.
:param lineSep: defines the line separator that should be used for parsing. If None is
set, it covers all ``\\r``, ``\\r\\n`` and ``\\n``.
+ :param samplingRatio: defines fraction of rows (when ``multiLine`` is ``false``) or fraction
--- End diff --
+1 (json -> JSON)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89692/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r183993973
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,67 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val ds = sampledTestData.coalesce(1)
+ ds.write.text(path.getAbsolutePath)
+ val readback = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
+ })
+ }
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
- assert(ds.schema == new StructType().add("f1", LongType))
- }
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val ds = sampledTestData.coalesce(1)
+ val readback = spark.read.option("samplingRatio", 0.1).json(ds)
+
+ assert(readback.schema == new StructType().add("f1", LongType))
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val ds = spark.range(0, 100, 1, 1).map(_.toString)
+
+ val errorMsg0 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", -1).json(ds)
+ }.getMessage
+ assert(errorMsg0.contains("samplingRatio (-1.0) should be greater than 0"))
- assert(ds.schema == new StructType().add("f1", LongType))
+ val errorMsg1 = intercept[IllegalArgumentException] {
+ spark.read.option("samplingRatio", 0).json(ds)
+ }.getMessage
+ assert(errorMsg1.contains("samplingRatio (0.0) should be greater than 0"))
+
+ val sampled = spark.read.option("samplingRatio", 10).json(ds)
+ assert(sampled.count() == ds.count())
+ }
+
+ test("SPARK-23849: sampling files for schema inferring in the multiLine mode") {
+ withTempDir { dir =>
+ Files.write(Paths.get(dir.getAbsolutePath, "0.json"),
+ """{"a":"a"}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ for (i <- 1 until 10) {
+ Files.write(Paths.get(dir.getAbsolutePath, s"$i.json"),
+ s"""{"a":$i}""".getBytes(StandardCharsets.UTF_8), StandardOpenOption.CREATE_NEW)
+ }
+ val files = (0 until 10).map { i =>
+ val hadoopConf = spark.sessionState.newHadoopConf()
+ val hdfsPath = new Path(Paths.get(dir.getAbsolutePath, s"$i.json").toString)
+ hdfsPath.getFileSystem(hadoopConf).getFileStatus(hdfsPath)
+ }
+ // The test uses the internal method because public API cannot guarantee order of files
+ // passed to the infer method. The order is changed between runs because the temporary
+ // folder has different path which leads to different order of file statuses returned
--- End diff --
Do you mean to pass a dir into the _infer()_ method instead of sequence of files?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89861/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by MaxGekk <gi...@git.apache.org>.
Github user MaxGekk commented on the issue:
https://github.com/apache/spark/pull/21056
@rxin May I ask you to look at the PR again.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21056#discussion_r181530121
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala ---
@@ -2128,38 +2128,60 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData {
}
}
- test("SPARK-23849: schema inferring touches less data if samplingRation < 1.0") {
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
+ val sampledTestData = (row: Row) => {
+ val value = row.getLong(0)
+ val predefinedSample = Set[Long](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
57, 62, 68, 72)
- withTempPath { path =>
- val writer = Files.newBufferedWriter(Paths.get(path.getAbsolutePath),
- StandardCharsets.UTF_8, StandardOpenOption.CREATE_NEW)
- for (i <- 0 until 100) {
- if (predefinedSample.contains(i)) {
- writer.write(s"""{"f1":${i.toString}}""" + "\n")
- } else {
- writer.write(s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n")
- }
- }
- writer.close()
+ if (predefinedSample.contains(value)) {
+ s"""{"f1":${value.toString}}"""
+ } else {
+ s"""{"f1":${(value.toDouble + 0.1).toString}}"""
+ }
+ }
- val ds = spark.read.option("samplingRatio", 0.1).json(path.getCanonicalPath)
+ test("SPARK-23849: schema inferring touches less data if samplingRatio < 1.0") {
+ // Set default values for the DataSource parameters to make sure
+ // that whole test file is mapped to only one partition. This will guarantee
+ // reliable sampling of the input file.
+ withSQLConf(
+ "spark.sql.files.maxPartitionBytes" -> (128 * 1024 * 1024).toString,
+ "spark.sql.files.openCostInBytes" -> (4 * 1024 * 1024).toString
+ )(withTempPath { path =>
+ val rdd = spark.sqlContext.range(0, 100, 1, 1).map(sampledTestData)
+ rdd.write.text(path.getAbsolutePath)
+
+ val ds = spark.read
+ .option("inferSchema", true)
+ .option("samplingRatio", 0.1)
+ .json(path.getCanonicalPath)
assert(ds.schema == new StructType().add("f1", LongType))
- }
+ })
}
- test("SPARK-23849: usage of samplingRation while parsing of dataset of strings") {
- val dstr = spark.sparkContext.parallelize(0 until 100, 1).map { i =>
- val predefinedSample = Set[Int](2, 8, 15, 27, 30, 34, 35, 37, 44, 46,
- 57, 62, 68, 72)
- if (predefinedSample.contains(i)) {
- s"""{"f1":${i.toString}}""" + "\n"
- } else {
- s"""{"f1":${(i.toDouble + 0.1).toString}}""" + "\n"
- }
- }.toDS()
- val ds = spark.read.option("samplingRatio", 0.1).json(dstr)
+ test("SPARK-23849: usage of samplingRatio while parsing a dataset of strings") {
+ val rdd = spark.sqlContext.range(0, 100, 1, 1).map(sampledTestData)
+ val ds = spark.read
+ .option("inferSchema", true)
+ .option("samplingRatio", 0.1)
+ .json(rdd)
assert(ds.schema == new StructType().add("f1", LongType))
}
+
+ test("SPARK-23849: samplingRatio is out of the range (0, 1.0]") {
+ val dstr = spark.sparkContext.parallelize(0 until 100, 1).map(_.toString).toDS()
--- End diff --
can you just use spark.range?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21056
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89285/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org