You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by xubo245 <gi...@git.apache.org> on 2018/02/12 03:38:31 UTC
[GitHub] spark pull request #20583: [CARBONDATA-23392][TEST] Add some test cases for ...
GitHub user xubo245 opened a pull request:
https://github.com/apache/spark/pull/20583
[CARBONDATA-23392][TEST] Add some test cases for images feature
## What changes were proposed in this pull request?
Add some test cases for images feature
## How was this patch tested?
Add some test cases in ImageSchemaSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/spark CARBONDATA23392_AddTestForImage
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20583.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20583
----
commit bc375112274d862de536841208d6e7cda151afe2
Author: xubo245 <60...@...>
Date: 2018-02-12T03:28:41Z
[CARBONDATA-23392][TEST] Add some test case for images feature
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87326/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167567793
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.schema("image").dataType == columnSchema, "data do not fit ImageSchema")
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
--- End diff --
I think these kinds of assertions end up too brittle. I might focus on asserting the message is about "sampleRatio" only or something
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/spark/pull/20583
Sorry, done.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [CARBONDATA-23392][TEST] Add some test cases for images ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87318/testReport)** for PR 20583 at commit [`bc37511`](https://github.com/apache/spark/commit/bc375112274d862de536841208d6e7cda151afe2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87319/testReport)** for PR 20583 at commit [`af269d7`](https://github.com/apache/spark/commit/af269d7a081a7169e73da3132f2e6eceeb3b1e4e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167481154
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio = 0") {
+ val df = readImages(imagePath, null, true, 3, true, 0.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: with sparkSession") {
+ val df = readImages(imagePath, sparkSession = spark, true, 3, true, 1.0, 0)
--- End diff --
It should be. I try it now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87323/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87323/testReport)** for PR 20583 at commit [`b67955a`](https://github.com/apache/spark/commit/b67955abf3a73adcca496cdc1557a866ec780ba9).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167479658
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio = 0") {
+ val df = readImages(imagePath, null, true, 3, true, 0.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: with sparkSession") {
+ val df = readImages(imagePath, sparkSession = spark, true, 3, true, 1.0, 0)
+ assert(df.count() === 7)
+ }
+
test("readImages partition test") {
val df = readImages(imagePath, null, true, 3, true, 1.0, 0)
assert(df.rdd.getNumPartitions === 3)
}
+ test("readImages partition test: < 0") {
+ val df = readImages(imagePath, null, true, -3, true, 1.0, 0)
+ assert(df.rdd.getNumPartitions === spark.sparkContext.defaultParallelism)
+ }
+
+ test("readImages partition test: = 0") {
--- End diff --
test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20583
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87326/testReport)** for PR 20583 at commit [`b67955a`](https://github.com/apache/spark/commit/b67955abf3a73adcca496cdc1557a866ec780ba9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20583
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/20583
Merged to master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87326/testReport)** for PR 20583 at commit [`b67955a`](https://github.com/apache/spark/commit/b67955abf3a73adcca496cdc1557a866ec780ba9).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87359/testReport)** for PR 20583 at commit [`4c18e23`](https://github.com/apache/spark/commit/4c18e232725f18156b56138471c52918d3fb83b3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87359/testReport)** for PR 20583 at commit [`4c18e23`](https://github.com/apache/spark/commit/4c18e232725f18156b56138471c52918d3fb83b3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by imatiach-msft <gi...@git.apache.org>.
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167470380
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio = 0") {
+ val df = readImages(imagePath, null, true, 3, true, 0.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: with sparkSession") {
+ val df = readImages(imagePath, sparkSession = spark, true, 3, true, 1.0, 0)
+ assert(df.count() === 7)
+ }
+
test("readImages partition test") {
val df = readImages(imagePath, null, true, 3, true, 1.0, 0)
assert(df.rdd.getNumPartitions === 3)
}
+ test("readImages partition test: < 0") {
+ val df = readImages(imagePath, null, true, -3, true, 1.0, 0)
+ assert(df.rdd.getNumPartitions === spark.sparkContext.defaultParallelism)
+ }
+
+ test("readImages partition test: = 0") {
+ val df = readImages(imagePath, null, true, 0, true, 1.0, 0)
+ assert(df.rdd.getNumPartitions != 0)
--- End diff --
the check here seems redundant because of the additional check below, maybe remove it (?)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by imatiach-msft <gi...@git.apache.org>.
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167471442
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio = 0") {
+ val df = readImages(imagePath, null, true, 3, true, 0.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: with sparkSession") {
+ val df = readImages(imagePath, sparkSession = spark, true, 3, true, 1.0, 0)
--- End diff --
it looks like this test is failing in the latest build? I'm not sure why, the error message says the count is 8. This test, which is similar, seems to have a count of 7 ("readImages count test"):
df = readImages(imagePath, null, true, -1, true, 1.0, 0)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87319/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87319/testReport)** for PR 20583 at commit [`af269d7`](https://github.com/apache/spark/commit/af269d7a081a7169e73da3132f2e6eceeb3b1e4e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87359/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87320/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20583
Thanks, @xubo245. Seems fine. cc @MrBago, @imatiach-msft, @jkbradley, @WeichenXu123 who I can directly think of for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [CARBONDATA-23392][TEST] Add some test cases for images ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20583
Shall we fix the title to `[SPARK-23392] ...`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87320/testReport)** for PR 20583 at commit [`a21de66`](https://github.com/apache/spark/commit/a21de669701ec0a985dcccf04e76bf942dd48ea0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87323/testReport)** for PR 20583 at commit [`b67955a`](https://github.com/apache/spark/commit/b67955abf3a73adcca496cdc1557a866ec780ba9).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167740107
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.schema("image").dataType == columnSchema, "data do not fit ImageSchema")
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
--- End diff --
Thanks, done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by imatiach-msft <gi...@git.apache.org>.
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167470275
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
--- End diff --
(optional) you can also validate here that the entry has the correct structure for an invalid image
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87320/testReport)** for PR 20583 at commit [`a21de66`](https://github.com/apache/spark/commit/a21de669701ec0a985dcccf04e76bf942dd48ea0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167545871
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio = 0") {
+ val df = readImages(imagePath, null, true, 3, true, 0.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: with sparkSession") {
+ val df = readImages(imagePath, sparkSession = spark, true, 3, true, 1.0, 0)
--- End diff --
Can you check it? This PR not merge into branch2.3: https://github.com/apache/spark/pull/20389
I fetch the code of branch 2.3 before.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by imatiach-msft <gi...@git.apache.org>.
Github user imatiach-msft commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167470547
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio = 0") {
+ val df = readImages(imagePath, null, true, 3, true, 0.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: with sparkSession") {
+ val df = readImages(imagePath, sparkSession = spark, true, 3, true, 1.0, 0)
+ assert(df.count() === 7)
--- End diff --
I think this should be spark.sparkContext.defaultParallelism instead of 7 (?)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167479566
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
--- End diff --
Ok, done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [CARBONDATA-23392][TEST] Add some test cases for images ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20583
**[Test build #87318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87318/testReport)** for PR 20583 at commit [`bc37511`](https://github.com/apache/spark/commit/bc375112274d862de536841208d6e7cda151afe2).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87318/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20583: [SPARK-23392][TEST] Add some test cases for images featu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20583
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20583: [SPARK-23392][TEST] Add some test cases for image...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20583#discussion_r167479724
--- Diff: mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala ---
@@ -65,11 +65,71 @@ class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
assert(count50 > 0 && count50 < countTotal)
}
+ test("readImages test: recursive = false") {
+ val df = readImages(imagePath, null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read jpg image") {
+ val df = readImages(imagePath + "/kittens/DP153539.jpg", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read png image") {
+ val df = readImages(imagePath + "/multi-channel/BGRA.png", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: read non image") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, true, 1.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: read non image and dropImageFailures is false") {
+ val df = readImages(imagePath + "/kittens/not-image.txt", null, false, 3, false, 1.0, 0)
+ assert(df.count() === 1)
+ }
+
+ test("readImages test: sampleRatio > 1") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, 1.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio < 0") {
+ val e = intercept[IllegalArgumentException] {
+ readImages(imagePath, null, true, 3, true, -0.1, 0)
+ }
+ assert(e.getMessage.equals("requirement failed: sampleRatio should be between 0 and 1"))
+ }
+
+ test("readImages test: sampleRatio = 0") {
+ val df = readImages(imagePath, null, true, 3, true, 0.0, 0)
+ assert(df.count() === 0)
+ }
+
+ test("readImages test: with sparkSession") {
+ val df = readImages(imagePath, sparkSession = spark, true, 3, true, 1.0, 0)
+ assert(df.count() === 7)
+ }
+
test("readImages partition test") {
val df = readImages(imagePath, null, true, 3, true, 1.0, 0)
assert(df.rdd.getNumPartitions === 3)
}
+ test("readImages partition test: < 0") {
+ val df = readImages(imagePath, null, true, -3, true, 1.0, 0)
+ assert(df.rdd.getNumPartitions === spark.sparkContext.defaultParallelism)
+ }
+
+ test("readImages partition test: = 0") {
+ val df = readImages(imagePath, null, true, 0, true, 1.0, 0)
+ assert(df.rdd.getNumPartitions != 0)
--- End diff --
Ok, done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org