You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/08/26 19:26:56 UTC
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSource suite validating data s...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/19060
[WIP][SQL] Add DataSource suite validating data sources limitations
## What changes were proposed in this pull request?
This test suite aims to test if underlying data sources supports Spark data values limits and PPD.
## How was this patch tested?
Pass the Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-DS-SUITE
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19060.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19060
----
commit 0bf9ad2ebff81f45eda9887d631cac5ddcc70d6c
Author: Dongjoon Hyun <do...@apache.org>
Date: 2017-08-26T19:24:46Z
[SPARK-XXX][SQL] Add DataSource suite validating data sources limitations
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
Today, I've re-considered this PR from the bottom and from the beginning.
To sum up, I realized that I've been wasting committer's review time. Especially, sorry for your effort, @gatorsmile !
As you recommended, I had better focus ono individual test suites more.
> So far, the coverage of OrcFilterSuite and ParquetFilterSuite are not good.
> They only have very basic checks. Could you improve them?
I'll close this PR for now. Thank you for giving me your advice!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81318/testReport)** for PR 19060 at commit [`4466325`](https://github.com/apache/spark/commit/44663258c529f0c426b777736309e5a82cf72d92).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81318/testReport)** for PR 19060 at commit [`4466325`](https://github.com/apache/spark/commit/44663258c529f0c426b777736309e5a82cf72d92).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r137066894
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
What we didn't agree is only **the corresponding tests**.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
Hi, @gatorsmile .
I believe I understand your advice correctly at this time.
Could you take a look at this `data source` verification PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81157/testReport)** for PR 19060 at commit [`16031ac`](https://github.com/apache/spark/commit/16031ac5771200ab9ecc47052e1b4c263fa1b8c0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
For predicate push-down, I ported ORC code already in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
For Parquet, I can find [TestInputOutputFormat.java](https://github.com/Parquet/parquet-mr/blob/master/parquet-hadoop/src/test/java/parquet/hadoop/example/TestInputOutputFormat.java). Parquet also has a test case which is very specific to the impl, too.
What do you mean by `e2e test case not specific to the impl.` exactly? Sorry, but could you provide an example?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81155/testReport)** for PR 19060 at commit [`0bf9ad2`](https://github.com/apache/spark/commit/0bf9ad2ebff81f45eda9887d631cac5ddcc70d6c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun closed the pull request at:
https://github.com/apache/spark/pull/19060
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
BTW, from the text-based format, I thought we can include JSON at the beginning. But, unfortunatly, I found that currently Spark use BIGINT for all numeric, DOUBLE for float/double, and STRING for DATE/TIMESTAMP/STRING. Simply, `checkValue` fails. I think we may cover Parquet and ORC only.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19060
The ORC test is very specific to the impl. No end-to-end test cases?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
Parquet is the same. We can use `unhandledFilters` for PPD.
I think that the others text-based data sources(TEXT/CSV/JSON) doesn't support PPD.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r137071948
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
So far, Parquet and ORC are the only built-in sources that support PPD. From the viewpoints of software development, we should make the test framework easily extensible.
Sorry, I do not think it covers what we need. First, to verify whether the underlying data source works properly, we need to have the corresponding (boundary) data (which are inserted to the sources), the predicates (which are using boundary values), and also need to ensure these predicates are pushed down and not evaluated by Spark.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19060
Does ORC/Parquet have the related test cases? Could we just port them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81298/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r137044001
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
I do not know why we did not turn it on by default. To me, to turn it on, we need to improve the coverage.
So far, the coverage of `OrcFilterSuite ` and `ParquetFilterSuite ` are not good. They only have very basic checks. Could you improve them?
For example, adding the boundary values for these predicate pushdown in both sides? We need to ensure whether the predicates are pushed down, executed in the underlying data sources, and the filters work properly (value comparison).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81155/testReport)** for PR 19060 at commit [`0bf9ad2`](https://github.com/apache/spark/commit/0bf9ad2ebff81f45eda9887d631cac5ddcc70d6c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
In ORC GitHub, up to my knowledge, this is the highest level.
```
val reader = new OrcInputFormat[OrcStruct]().createRecordReader(split, attemptContext)
... reader.nextKeyValue()
... reader.getCurrentValue
... row.getFieldValue(0).asInstanceOf[IntWritable].get
```
Actually, I have one idea. If we support [unhandledFilters](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala#L237) in data source testing by SQLConf, we can do in Spark level like *value limit* case. How do you think about that? May I try that way?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r136912257
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
This has been tested in `OrcFilterSuite` and `ParquetFilterSuite`, right? What is the goal of this test case?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r137052603
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
Parquet is the default format. It is being used by most our Spark users. We already got many related JIRA issues and then fixed/blocked them. You were also involved in some of these PRs.
To avoid repeating the same issue in ORC, we should improve the coverage before turning on ORC predicate pushdown by default.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r136475026
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.io.{IntWritable, NullWritable, Text}
+import org.apache.hadoop.mapred.JobConf
+import org.apache.hadoop.mapreduce._
+import org.apache.hadoop.mapreduce.lib.input.FileSplit
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.orc.{OrcConf, TypeDescription}
+import org.apache.orc.mapred.OrcStruct
+import org.apache.orc.mapreduce.{OrcInputFormat, OrcOutputFormat}
+import org.apache.orc.storage.ql.io.sarg.{PredicateLeaf, SearchArgumentFactory}
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits
+ * - Predicate Push Down
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
--- End diff --
`json` and `csv` is considered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSource suite validating data sources ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
Hi, @gatorsmile .
This is the poc on new ORC data source. Please note that old ORC uses different packages and apis.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r137060959
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
I have different philosophy. I worked for mission-critical enterprise software. I do not trust anybody's codes unless they can pass the corresponding tests.
The initial ORC PPD does not have enough test coverage. To enable it by default, we should improve the test coverage now. The new test case framework for PPD should cover all the sources. We can have source-speicific ones, if it is not applicable to the other sources.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r137064595
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
Yep. This PR is based on the following your guidelines and I wish that it will continue to evolve. :)
- Handle all data sources (For value ranges, I tested on Spark min/max instead of data source min/max)
- Use high-level end-to-end test case
For PPD, Parquet and ORC are the only ones supporting PPD, aren't they?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81365/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81318/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r136932032
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
`OrcFilterSuite` and `ParquetFilterSuite` does not test `stripe` level filtering in a single file. It only filters out the file-level filtering, doesn't it?
This test suite is designed to prove that PPD on ORC data source doesn't filter out wrongly. Since ORC does not filter row-by-row, this PR check the all values of the returned stripe. **All values in the stripe exist correctly**.
IIRC, this is what you requested in [SPARK-21783 Turn on ORC filter push-down by default](https://github.com/apache/spark/pull/18991). If `OrcFilterSuite` has been complete, I'm wondering what is the reason to block turning on `ORC PPD` by default.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
If you agree, I will try to write more code here as POC.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r137057897
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
1. Every JIRA issue in Apache Spark should have a test case to be resolved. If then, we can focus on ORC test case parity for Parquet. Adding more test cases is beyond of that scope.
> We already got many related JIRA issues and then fixed/blocked them.
2. Yes. ORC is not a default format. Even worse, although a user try to use '.orc(...)`, PPD is not used by default, too. We cannot find any issues on PPD. It doesn't form a virtuous cycle at all. To be honest with you, I trust the efforts on Apache ORC community.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
For ORC, I was able to find some random-generated float/double tests, but there is no value-limit test. For Parquet, I'm not sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81157/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19060: [WIP][SQL] Add DataSourceSuite validating data so...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19060#discussion_r137050287
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/sources/DataSourceSuite.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import java.sql.{Date, Timestamp}
+
+import org.apache.orc.OrcConf
+
+import org.apache.spark.sql.{Dataset, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+/**
+ * Data Source qualification as Apache Spark Data Sources.
+ * - Apache Spark Data Type Value Limits: CSV, JSON, ORC, Parquet
+ * - Predicate Push Down: ORC
+ */
+class DataSourceSuite
+ extends QueryTest
+ with SQLTestUtils
+ with TestHiveSingleton {
+
+ import testImplicits._
+
+ var df: Dataset[Row] = _
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ spark.conf.set("spark.sql.session.timeZone", "GMT")
+
+ df = ((
+ false,
+ true,
+ Byte.MinValue,
+ Byte.MaxValue,
+ Short.MinValue,
+ Short.MaxValue,
+ Int.MinValue,
+ Int.MaxValue,
+ Long.MinValue,
+ Long.MaxValue,
+ Float.MinValue,
+ Float.MaxValue,
+ Double.MinValue,
+ Double.MaxValue,
+ Date.valueOf("0001-01-01"),
+ Date.valueOf("9999-12-31"),
+ new Timestamp(-62135769600000L), // 0001-01-01 00:00:00.000
+ new Timestamp(253402300799999L) // 9999-12-31 23:59:59.999
+ ) :: Nil).toDF()
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.conf.unset("spark.sql.session.timeZone")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ Seq("parquet", "orc", "json", "csv").foreach { dataSource =>
+ test(s"$dataSource - data type value limit") {
+ withTempPath { dir =>
+ df.write.format(dataSource).save(dir.getCanonicalPath)
+
+ // Use the same schema for saving/loading
+ checkAnswer(
+ spark.read.format(dataSource).schema(df.schema).load(dir.getCanonicalPath),
+ df)
+
+ // Use schema inference, but skip text-based format due to its limitation
+ if (Seq("parquet", "orc").contains(dataSource)) {
+ withTable("tab1") {
+ sql(s"CREATE TABLE tab1 USING $dataSource LOCATION '${dir.toURI}'")
+ checkAnswer(sql(s"SELECT ${df.schema.fieldNames.mkString(",")} FROM tab1"), df)
+ }
+ }
+ }
+ }
+ }
+
+ Seq("orc").foreach { dataSource =>
+ test(s"$dataSource - predicate push down") {
+ withSQLConf(
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ withTempPath { dir =>
+ // write 4000 rows with the integer and the string in a single orc file with stride 1000
+ spark
+ .range(4000)
+ .map(i => (i, s"$i"))
+ .toDF("i", "s")
+ .repartition(1)
+ .write
+ .option(OrcConf.ROW_INDEX_STRIDE.getAttribute, 1000)
+ // TODO: Add Parquet option, too.
+ .format(dataSource)
+ .save(dir.getCanonicalPath)
+
+ val df = spark.read.format(dataSource).load(dir.getCanonicalPath)
+ .where(s"i BETWEEN 1500 AND 1999")
--- End diff --
I agree with you on improving the coverage, @gatorsmile . So, I've started this and have been trying to improve this in order to find a way what you want.
Given that all the situations are the same with Parquet and ORC, what I suggested on #18991 is to turn on ORC PPD by default at least before 2.3.0 release. How do you think about that?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81298/testReport)** for PR 19060 at commit [`104f24c`](https://github.com/apache/spark/commit/104f24c9ad0743dc7c6329b4c0dde902e8e87de6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81365/testReport)** for PR 19060 at commit [`fb72c89`](https://github.com/apache/spark/commit/fb72c89478149da99bd7ac402257ab7468156f9d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
Hi, @gatorsmile .
I make it to use more higher level in `test("orc - predicate push down"`.
How do you think about this approach?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81157/testReport)** for PR 19060 at commit [`16031ac`](https://github.com/apache/spark/commit/16031ac5771200ab9ecc47052e1b4c263fa1b8c0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19060
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81155/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81365/testReport)** for PR 19060 at commit [`fb72c89`](https://github.com/apache/spark/commit/fb72c89478149da99bd7ac402257ab7468156f9d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19060
I mean, how about Parquet and the others? Do they have the e2e test cases in their projects?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19060
Previous parquet link is broken. The official one is https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/test/java/org/apache/parquet/hadoop/example/TestInputOutputFormat.java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19060
How about Parquet and the others?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19060: [WIP][SQL] Add DataSourceSuite validating data sources l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19060
**[Test build #81298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81298/testReport)** for PR 19060 at commit [`104f24c`](https://github.com/apache/spark/commit/104f24c9ad0743dc7c6329b4c0dde902e8e87de6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org