You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by me...@apache.org on 2018/09/05 18:59:04 UTC
spark git commit: [SPARK-22666][ML][SQL] Spark datasource for image
format
Repository: spark
Updated Branches:
refs/heads/master c66eef844 -> 925449283
[SPARK-22666][ML][SQL] Spark datasource for image format
## What changes were proposed in this pull request?
Implement an image schema datasource.
This image datasource support:
- partition discovery (loading partitioned images)
- dropImageFailures (the same behavior with `ImageSchema.readImage`)
- path wildcard matching (the same behavior with `ImageSchema.readImage`)
- loading recursively from directory (different from `ImageSchema.readImage`, but use such path: `/path/to/dir/**`)
This datasource **NOT** support:
- specify `numPartitions` (it will be determined by datasource automatically)
- sampling (you can use `df.sample` later but the sampling operator won't be pushdown to datasource)
## How was this patch tested?
Unit tests.
## Benchmark
I benchmark and compare the cost time between old `ImageSchema.read` API and my image datasource.
**cluster**: 4 nodes, each with 64GB memory, 8 cores CPU
**test dataset**: Flickr8k_Dataset (about 8091 images)
**time cost**:
- My image datasource time (automatically generate 258 partitions): 38.04s
- `ImageSchema.read` time (set 16 partitions): 68.4s
- `ImageSchema.read` time (set 258 partitions): 90.6s
**time cost when increase image number by double (clone Flickr8k_Dataset and loads double number images)**:
- My image datasource time (automatically generate 515 partitions): 95.4s
- `ImageSchema.read` (set 32 partitions): 109s
- `ImageSchema.read` (set 515 partitions): 105s
So we can see that my image datasource implementation (this PR) bring some performance improvement compared against old`ImageSchema.read` API.
Closes #22328 from WeichenXu123/image_datasource.
Authored-by: WeichenXu <we...@databricks.com>
Signed-off-by: Xiangrui Meng <me...@databricks.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/92544928
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/92544928
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/92544928
Branch: refs/heads/master
Commit: 925449283dcaef80e0f77e60aea6ef988bd697b4
Parents: c66eef8
Author: WeichenXu <we...@databricks.com>
Authored: Wed Sep 5 11:59:00 2018 -0700
Committer: Xiangrui Meng <me...@databricks.com>
Committed: Wed Sep 5 11:59:00 2018 -0700
----------------------------------------------------------------------
.../images/kittens/29.5.a_b_EGDP022204.jpg | Bin 27295 -> 0 bytes
data/mllib/images/kittens/54893.jpg | Bin 35914 -> 0 bytes
data/mllib/images/kittens/DP153539.jpg | Bin 26354 -> 0 bytes
data/mllib/images/kittens/DP802813.jpg | Bin 30432 -> 0 bytes
data/mllib/images/kittens/not-image.txt | 1 -
data/mllib/images/multi-channel/BGRA.png | Bin 683 -> 0 bytes
.../images/multi-channel/BGRA_alpha_60.png | Bin 747 -> 0 bytes
data/mllib/images/multi-channel/chr30.4.184.jpg | Bin 59472 -> 0 bytes
data/mllib/images/multi-channel/grayscale.jpg | Bin 36728 -> 0 bytes
.../origin/kittens/29.5.a_b_EGDP022204.jpg | Bin 0 -> 27295 bytes
data/mllib/images/origin/kittens/54893.jpg | Bin 0 -> 35914 bytes
data/mllib/images/origin/kittens/DP153539.jpg | Bin 0 -> 26354 bytes
data/mllib/images/origin/kittens/DP802813.jpg | Bin 0 -> 30432 bytes
data/mllib/images/origin/kittens/not-image.txt | 1 +
data/mllib/images/origin/license.txt | 13 ++
data/mllib/images/origin/multi-channel/BGRA.png | Bin 0 -> 683 bytes
.../origin/multi-channel/BGRA_alpha_60.png | Bin 0 -> 747 bytes
.../images/origin/multi-channel/chr30.4.184.jpg | Bin 0 -> 59472 bytes
.../images/origin/multi-channel/grayscale.jpg | Bin 0 -> 36728 bytes
.../date=2018-01/29.5.a_b_EGDP022204.jpg | Bin 0 -> 27295 bytes
.../cls=kittens/date=2018-01/not-image.txt | 1 +
.../cls=kittens/date=2018-02/54893.jpg | Bin 0 -> 35914 bytes
.../cls=kittens/date=2018-02/DP153539.jpg | Bin 0 -> 26354 bytes
.../cls=kittens/date=2018-02/DP802813.jpg | Bin 0 -> 30432 bytes
.../cls=multichannel/date=2018-01/BGRA.png | Bin 0 -> 683 bytes
.../date=2018-01/BGRA_alpha_60.png | Bin 0 -> 747 bytes
.../date=2018-02/chr30.4.184.jpg | Bin 0 -> 59472 bytes
.../cls=multichannel/date=2018-02/grayscale.jpg | Bin 0 -> 36728 bytes
....apache.spark.sql.sources.DataSourceRegister | 1 +
.../spark/ml/source/image/ImageDataSource.scala | 53 +++++++++
.../spark/ml/source/image/ImageFileFormat.scala | 100 ++++++++++++++++
.../spark/ml/source/image/ImageOptions.scala | 32 +++++
.../spark/ml/image/ImageSchemaSuite.scala | 2 +-
.../ml/source/image/ImageFileFormatSuite.scala | 119 +++++++++++++++++++
python/pyspark/ml/image.py | 2 +-
python/pyspark/ml/tests.py | 4 +-
36 files changed, 324 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/kittens/29.5.a_b_EGDP022204.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/kittens/29.5.a_b_EGDP022204.jpg b/data/mllib/images/kittens/29.5.a_b_EGDP022204.jpg
deleted file mode 100644
index 435e7df..0000000
Binary files a/data/mllib/images/kittens/29.5.a_b_EGDP022204.jpg and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/kittens/54893.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/kittens/54893.jpg b/data/mllib/images/kittens/54893.jpg
deleted file mode 100644
index 825630c..0000000
Binary files a/data/mllib/images/kittens/54893.jpg and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/kittens/DP153539.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/kittens/DP153539.jpg b/data/mllib/images/kittens/DP153539.jpg
deleted file mode 100644
index 571efe9..0000000
Binary files a/data/mllib/images/kittens/DP153539.jpg and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/kittens/DP802813.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/kittens/DP802813.jpg b/data/mllib/images/kittens/DP802813.jpg
deleted file mode 100644
index 2d12359..0000000
Binary files a/data/mllib/images/kittens/DP802813.jpg and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/kittens/not-image.txt
----------------------------------------------------------------------
diff --git a/data/mllib/images/kittens/not-image.txt b/data/mllib/images/kittens/not-image.txt
deleted file mode 100644
index 283e5e9..0000000
--- a/data/mllib/images/kittens/not-image.txt
+++ /dev/null
@@ -1 +0,0 @@
-not an image
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/multi-channel/BGRA.png
----------------------------------------------------------------------
diff --git a/data/mllib/images/multi-channel/BGRA.png b/data/mllib/images/multi-channel/BGRA.png
deleted file mode 100644
index a944c6c..0000000
Binary files a/data/mllib/images/multi-channel/BGRA.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/multi-channel/BGRA_alpha_60.png
----------------------------------------------------------------------
diff --git a/data/mllib/images/multi-channel/BGRA_alpha_60.png b/data/mllib/images/multi-channel/BGRA_alpha_60.png
deleted file mode 100644
index 913637c..0000000
Binary files a/data/mllib/images/multi-channel/BGRA_alpha_60.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/multi-channel/chr30.4.184.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/multi-channel/chr30.4.184.jpg b/data/mllib/images/multi-channel/chr30.4.184.jpg
deleted file mode 100644
index 7068b97..0000000
Binary files a/data/mllib/images/multi-channel/chr30.4.184.jpg and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/multi-channel/grayscale.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/multi-channel/grayscale.jpg b/data/mllib/images/multi-channel/grayscale.jpg
deleted file mode 100644
index 621cdd1..0000000
Binary files a/data/mllib/images/multi-channel/grayscale.jpg and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg b/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg
new file mode 100644
index 0000000..435e7df
Binary files /dev/null and b/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/kittens/54893.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/kittens/54893.jpg b/data/mllib/images/origin/kittens/54893.jpg
new file mode 100644
index 0000000..825630c
Binary files /dev/null and b/data/mllib/images/origin/kittens/54893.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/kittens/DP153539.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/kittens/DP153539.jpg b/data/mllib/images/origin/kittens/DP153539.jpg
new file mode 100644
index 0000000..571efe9
Binary files /dev/null and b/data/mllib/images/origin/kittens/DP153539.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/kittens/DP802813.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/kittens/DP802813.jpg b/data/mllib/images/origin/kittens/DP802813.jpg
new file mode 100644
index 0000000..2d12359
Binary files /dev/null and b/data/mllib/images/origin/kittens/DP802813.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/kittens/not-image.txt
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/kittens/not-image.txt b/data/mllib/images/origin/kittens/not-image.txt
new file mode 100644
index 0000000..283e5e9
--- /dev/null
+++ b/data/mllib/images/origin/kittens/not-image.txt
@@ -0,0 +1 @@
+not an image
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/license.txt
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/license.txt b/data/mllib/images/origin/license.txt
new file mode 100644
index 0000000..052f302
--- /dev/null
+++ b/data/mllib/images/origin/license.txt
@@ -0,0 +1,13 @@
+The images in the folder "kittens" are under the creative commons CC0 license, or no rights reserved:
+https://creativecommons.org/share-your-work/public-domain/cc0/
+The images are taken from:
+https://ccsearch.creativecommons.org/image/detail/WZnbJSJ2-dzIDiuUUdto3Q==
+https://ccsearch.creativecommons.org/image/detail/_TlKu_rm_QrWlR0zthQTXA==
+https://ccsearch.creativecommons.org/image/detail/OPNnHJb6q37rSZ5o_L5JHQ==
+https://ccsearch.creativecommons.org/image/detail/B2CVP_j5KjwZm7UAVJ3Hvw==
+
+The chr30.4.184.jpg and grayscale.jpg images are also under the CC0 license, taken from:
+https://ccsearch.creativecommons.org/image/detail/8eO_qqotBfEm2UYxirLntw==
+
+The image under "multi-channel" directory is under the CC BY-SA 4.0 license cropped from:
+https://en.wikipedia.org/wiki/Alpha_compositing#/media/File:Hue_alpha_falloff.png
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/multi-channel/BGRA.png
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/multi-channel/BGRA.png b/data/mllib/images/origin/multi-channel/BGRA.png
new file mode 100644
index 0000000..a944c6c
Binary files /dev/null and b/data/mllib/images/origin/multi-channel/BGRA.png differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/multi-channel/BGRA_alpha_60.png
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/multi-channel/BGRA_alpha_60.png b/data/mllib/images/origin/multi-channel/BGRA_alpha_60.png
new file mode 100644
index 0000000..913637c
Binary files /dev/null and b/data/mllib/images/origin/multi-channel/BGRA_alpha_60.png differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/multi-channel/chr30.4.184.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/multi-channel/chr30.4.184.jpg b/data/mllib/images/origin/multi-channel/chr30.4.184.jpg
new file mode 100644
index 0000000..7068b97
Binary files /dev/null and b/data/mllib/images/origin/multi-channel/chr30.4.184.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/origin/multi-channel/grayscale.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/origin/multi-channel/grayscale.jpg b/data/mllib/images/origin/multi-channel/grayscale.jpg
new file mode 100644
index 0000000..621cdd1
Binary files /dev/null and b/data/mllib/images/origin/multi-channel/grayscale.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg b/data/mllib/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg
new file mode 100644
index 0000000..435e7df
Binary files /dev/null and b/data/mllib/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=kittens/date=2018-01/not-image.txt
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=kittens/date=2018-01/not-image.txt b/data/mllib/images/partitioned/cls=kittens/date=2018-01/not-image.txt
new file mode 100644
index 0000000..283e5e9
--- /dev/null
+++ b/data/mllib/images/partitioned/cls=kittens/date=2018-01/not-image.txt
@@ -0,0 +1 @@
+not an image
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=kittens/date=2018-02/54893.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=kittens/date=2018-02/54893.jpg b/data/mllib/images/partitioned/cls=kittens/date=2018-02/54893.jpg
new file mode 100644
index 0000000..825630c
Binary files /dev/null and b/data/mllib/images/partitioned/cls=kittens/date=2018-02/54893.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg b/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg
new file mode 100644
index 0000000..571efe9
Binary files /dev/null and b/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg b/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg
new file mode 100644
index 0000000..2d12359
Binary files /dev/null and b/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA.png
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA.png b/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA.png
new file mode 100644
index 0000000..a944c6c
Binary files /dev/null and b/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA.png differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png b/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png
new file mode 100644
index 0000000..913637c
Binary files /dev/null and b/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg b/data/mllib/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg
new file mode 100644
index 0000000..7068b97
Binary files /dev/null and b/data/mllib/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/data/mllib/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg
----------------------------------------------------------------------
diff --git a/data/mllib/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg b/data/mllib/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg
new file mode 100644
index 0000000..621cdd1
Binary files /dev/null and b/data/mllib/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg differ
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/mllib/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
----------------------------------------------------------------------
diff --git a/mllib/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister b/mllib/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
index a865cbe..a7dfd2d 100644
--- a/mllib/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
+++ b/mllib/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
@@ -1 +1,2 @@
org.apache.spark.ml.source.libsvm.LibSVMFileFormat
+org.apache.spark.ml.source.image.ImageFileFormat
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageDataSource.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageDataSource.scala b/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageDataSource.scala
new file mode 100644
index 0000000..a111c95
--- /dev/null
+++ b/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageDataSource.scala
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.source.image
+
+/**
+ * `image` package implements Spark SQL data source API for loading image data as `DataFrame`.
+ * The loaded `DataFrame` has one `StructType` column: `image`.
+ * The schema of the `image` column is:
+ * - origin: String (represents the file path of the image)
+ * - height: Int (height of the image)
+ * - width: Int (width of the image)
+ * - nChannels: Int (number of the image channels)
+ * - mode: Int (OpenCV-compatible type)
+ * - data: BinaryType (Image bytes in OpenCV-compatible order: row-wise BGR in most cases)
+ *
+ * To use image data source, you need to set "image" as the format in `DataFrameReader` and
+ * optionally specify the data source options, for example:
+ * {{{
+ * // Scala
+ * val df = spark.read.format("image")
+ * .option("dropInvalid", true)
+ * .load("data/mllib/images/partitioned")
+ *
+ * // Java
+ * Dataset<Row> df = spark.read().format("image")
+ * .option("dropInvalid", true)
+ * .load("data/mllib/images/partitioned");
+ * }}}
+ *
+ * Image data source supports the following options:
+ * - "dropInvalid": Whether to drop the files that are not valid images from the result.
+ *
+ * @note This IMAGE data source does not support saving images to files.
+ *
+ * @note This class is public for documentation purpose. Please don't use this class directly.
+ * Rather, use the data source API as illustrated above.
+ */
+class ImageDataSource private() {}
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala b/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala
new file mode 100644
index 0000000..c332144
--- /dev/null
+++ b/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageFileFormat.scala
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.source.image
+
+import com.google.common.io.{ByteStreams, Closeables}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.hadoop.mapreduce.Job
+
+import org.apache.spark.ml.image.ImageSchema
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, UnsafeRow}
+import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
+import org.apache.spark.sql.execution.datasources.{DataSource, FileFormat, OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.sources.{DataSourceRegister, Filter}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.SerializableConfiguration
+
+private[image] class ImageFileFormat extends FileFormat with DataSourceRegister {
+
+ override def inferSchema(
+ sparkSession: SparkSession,
+ options: Map[String, String],
+ files: Seq[FileStatus]): Option[StructType] = Some(ImageSchema.imageSchema)
+
+ override def prepareWrite(
+ sparkSession: SparkSession,
+ job: Job,
+ options: Map[String, String],
+ dataSchema: StructType): OutputWriterFactory = {
+ throw new UnsupportedOperationException("Write is not supported for image data source")
+ }
+
+ override def shortName(): String = "image"
+
+ override protected def buildReader(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ options: Map[String, String],
+ hadoopConf: Configuration): (PartitionedFile) => Iterator[InternalRow] = {
+ assert(
+ requiredSchema.length <= 1,
+ "Image data source only produces a single data column named \"image\".")
+
+ val broadcastedHadoopConf =
+ sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf))
+
+ val imageSourceOptions = new ImageOptions(options)
+
+ (file: PartitionedFile) => {
+ val emptyUnsafeRow = new UnsafeRow(0)
+ if (!imageSourceOptions.dropInvalid && requiredSchema.isEmpty) {
+ Iterator(emptyUnsafeRow)
+ } else {
+ val origin = file.filePath
+ val path = new Path(origin)
+ val fs = path.getFileSystem(broadcastedHadoopConf.value.value)
+ val stream = fs.open(path)
+ val bytes = try {
+ ByteStreams.toByteArray(stream)
+ } finally {
+ Closeables.close(stream, true)
+ }
+ val resultOpt = ImageSchema.decode(origin, bytes)
+ val filteredResult = if (imageSourceOptions.dropInvalid) {
+ resultOpt.toIterator
+ } else {
+ Iterator(resultOpt.getOrElse(ImageSchema.invalidImageRow(origin)))
+ }
+
+ if (requiredSchema.isEmpty) {
+ filteredResult.map(_ => emptyUnsafeRow)
+ } else {
+ val converter = RowEncoder(requiredSchema)
+ filteredResult.map(row => converter.toRow(row))
+ }
+ }
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageOptions.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageOptions.scala b/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageOptions.scala
new file mode 100644
index 0000000..7ff1969
--- /dev/null
+++ b/mllib/src/main/scala/org/apache/spark/ml/source/image/ImageOptions.scala
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.source.image
+
+import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
+
+private[image] class ImageOptions(
+ @transient private val parameters: CaseInsensitiveMap[String]) extends Serializable {
+
+ def this(parameters: Map[String, String]) = this(CaseInsensitiveMap(parameters))
+
+ /**
+ * Whether to drop invalid images. If true, invalid images will be removed, otherwise
+ * invalid images will be returned with empty data and all other field filled with `-1`.
+ */
+ val dropInvalid = parameters.getOrElse("dropInvalid", "false").toBoolean
+}
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala
----------------------------------------------------------------------
diff --git a/mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala b/mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala
index 527b3f8..e16ec90 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/image/ImageSchemaSuite.scala
@@ -28,7 +28,7 @@ import org.apache.spark.sql.types._
class ImageSchemaSuite extends SparkFunSuite with MLlibTestSparkContext {
// Single column of images named "image"
- private lazy val imagePath = "../data/mllib/images"
+ private lazy val imagePath = "../data/mllib/images/origin"
test("Smoke test: create basic ImageSchema dataframe") {
val origin = "path"
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala
----------------------------------------------------------------------
diff --git a/mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala b/mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala
new file mode 100644
index 0000000..1a6a8d6
--- /dev/null
+++ b/mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.source.image
+
+import java.nio.file.Paths
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.image.ImageSchema._
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.functions.{col, substring_index}
+
+class ImageFileFormatSuite extends SparkFunSuite with MLlibTestSparkContext {
+
+ // Single column of images named "image"
+ private lazy val imagePath = "../data/mllib/images/partitioned"
+
+ test("image datasource count test") {
+ val df1 = spark.read.format("image").load(imagePath)
+ assert(df1.count === 9)
+
+ val df2 = spark.read.format("image").option("dropInvalid", true).load(imagePath)
+ assert(df2.count === 8)
+ }
+
+ test("image datasource test: read jpg image") {
+ val df = spark.read.format("image").load(imagePath + "/cls=kittens/date=2018-02/DP153539.jpg")
+ assert(df.count() === 1)
+ }
+
+ test("image datasource test: read png image") {
+ val df = spark.read.format("image").load(imagePath + "/cls=multichannel/date=2018-01/BGRA.png")
+ assert(df.count() === 1)
+ }
+
+ test("image datasource test: read non image") {
+ val filePath = imagePath + "/cls=kittens/date=2018-01/not-image.txt"
+ val df = spark.read.format("image").option("dropInvalid", true)
+ .load(filePath)
+ assert(df.count() === 0)
+
+ val df2 = spark.read.format("image").option("dropInvalid", false)
+ .load(filePath)
+ assert(df2.count() === 1)
+ val result = df2.head()
+ assert(result === invalidImageRow(
+ Paths.get(filePath).toAbsolutePath().normalize().toUri().toString))
+ }
+
+ test("image datasource partition test") {
+ val result = spark.read.format("image")
+ .option("dropInvalid", true).load(imagePath)
+ .select(substring_index(col("image.origin"), "/", -1).as("origin"), col("cls"), col("date"))
+ .collect()
+
+ assert(Set(result: _*) === Set(
+ Row("29.5.a_b_EGDP022204.jpg", "kittens", "2018-01"),
+ Row("54893.jpg", "kittens", "2018-02"),
+ Row("DP153539.jpg", "kittens", "2018-02"),
+ Row("DP802813.jpg", "kittens", "2018-02"),
+ Row("BGRA.png", "multichannel", "2018-01"),
+ Row("BGRA_alpha_60.png", "multichannel", "2018-01"),
+ Row("chr30.4.184.jpg", "multichannel", "2018-02"),
+ Row("grayscale.jpg", "multichannel", "2018-02")
+ ))
+ }
+
+ // Images with the different number of channels
+ test("readImages pixel values test") {
+ val images = spark.read.format("image").option("dropInvalid", true)
+ .load(imagePath + "/cls=multichannel/").collect()
+
+ val firstBytes20Set = images.map { rrow =>
+ val row = rrow.getAs[Row]("image")
+ val filename = Paths.get(getOrigin(row)).getFileName().toString()
+ val mode = getMode(row)
+ val bytes20 = getData(row).slice(0, 20).toList
+ filename -> Tuple2(mode, bytes20) // Cannot remove `Tuple2`, otherwise `->` operator
+ // will match 2 arguments
+ }.toSet
+
+ assert(firstBytes20Set === expectedFirstBytes20Set)
+ }
+
+ // number of channels and first 20 bytes of OpenCV representation
+ // - default representation for 3-channel RGB images is BGR row-wise:
+ // (B00, G00, R00, B10, G10, R10, ...)
+ // - default representation for 4-channel RGB images is BGRA row-wise:
+ // (B00, G00, R00, A00, B10, G10, R10, A10, ...)
+ private val expectedFirstBytes20Set = Set(
+ "grayscale.jpg" ->
+ ((0, List[Byte](-2, -33, -61, -60, -59, -59, -64, -59, -66, -67, -73, -73, -62,
+ -57, -60, -63, -53, -49, -55, -69))),
+ "chr30.4.184.jpg" -> ((16,
+ List[Byte](-9, -3, -1, -43, -32, -28, -75, -60, -57, -78, -59, -56, -74, -59, -57,
+ -71, -58, -56, -73, -64))),
+ "BGRA.png" -> ((24,
+ List[Byte](-128, -128, -8, -1, -128, -128, -8, -1, -128,
+ -128, -8, -1, 127, 127, -9, -1, 127, 127, -9, -1))),
+ "BGRA_alpha_60.png" -> ((24,
+ List[Byte](-128, -128, -8, 60, -128, -128, -8, 60, -128,
+ -128, -8, 60, 127, 127, -9, 60, 127, 127, -9, 60)))
+ )
+}
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/python/pyspark/ml/image.py
----------------------------------------------------------------------
diff --git a/python/pyspark/ml/image.py b/python/pyspark/ml/image.py
index 5f0c57e..ef6785b 100644
--- a/python/pyspark/ml/image.py
+++ b/python/pyspark/ml/image.py
@@ -216,7 +216,7 @@ class _ImageSchema(object):
:return: a :class:`DataFrame` with a single column of "images",
see ImageSchema for details.
- >>> df = ImageSchema.readImages('data/mllib/images/kittens', recursive=True)
+ >>> df = ImageSchema.readImages('data/mllib/images/origin/kittens', recursive=True)
>>> df.count()
5
http://git-wip-us.apache.org/repos/asf/spark/blob/92544928/python/pyspark/ml/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/ml/tests.py b/python/pyspark/ml/tests.py
index 625d992..821e037 100755
--- a/python/pyspark/ml/tests.py
+++ b/python/pyspark/ml/tests.py
@@ -2186,7 +2186,7 @@ class FPGrowthTests(SparkSessionTestCase):
class ImageReaderTest(SparkSessionTestCase):
def test_read_images(self):
- data_path = 'data/mllib/images/kittens'
+ data_path = 'data/mllib/images/origin/kittens'
df = ImageSchema.readImages(data_path, recursive=True, dropImageFailures=True)
self.assertEqual(df.count(), 4)
first_row = df.take(1)[0][0]
@@ -2253,7 +2253,7 @@ class ImageReaderTest2(PySparkTestCase):
def test_read_images_multiple_times(self):
# This test case is to check if `ImageSchema.readImages` tries to
# initiate Hive client multiple times. See SPARK-22651.
- data_path = 'data/mllib/images/kittens'
+ data_path = 'data/mllib/images/origin/kittens'
ImageSchema.readImages(data_path, recursive=True, dropImageFailures=True)
ImageSchema.readImages(data_path, recursive=True, dropImageFailures=True)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org