You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/19 02:05:57 UTC
[GitHub] [spark] bersprockets opened a new pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
bersprockets opened a new pull request #32969:
URL: https://github.com/apache/spark/pull/32969
### What changes were proposed in this pull request?
When creating a record writer in an AvroDeserializer, or creating a struct converter in an AvroSerializer, look up Avro fields using a map rather than scanning the entire list of Avro fields.
### Why are the changes needed?
A query against an Avro table can be quite slow when all are true:
* There are many columns in the Avro file
* The query contains a wide projection
* There are many splits in the input
* Some of the splits are read serially (e.g., less executors than there are tasks)
A write to an Avro table can be quite slow when all are true:
* There are many columns in the new rows
* The operation is creating many files
For example, a single-threaded query against a 6000 column Avro data set with 50K rows and 20 files takes less than a minute with Spark 3.0.1 but over 7 minutes with Spark 3.2.0-SNAPSHOT.
This PR restores the faster time.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
* Ran existing unit tests
* Added new unit tests
* Added new benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #32969:
URL: https://github.com/apache/spark/pull/32969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r656355498
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
+
+ /**
+ * Extract a single field from the contained avro schema which has the desired field name,
+ * performing the matching with proper case sensitivity according to SQLConf.resolver.
+ *
+ * @param name The name of the field to search for.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ def getFieldByName(name: String): Option[Schema.Field] = {
+
+ // get candidates, ignoring case of field name
+ val candidates = schemaMap.get(name.toLowerCase(Locale.ROOT))
+ .getOrElse(Seq.empty[Schema.Field])
+
+ // search candidates, taking into account case sensitivity settings
+ candidates.filter(f => SQLConf.get.resolver(f.name(), name)) match {
+ case Seq(avroField) => Some(avroField)
Review comment:
I feel the current approach is a bit simpler (more of the code is shared between case-sensitive vs. insensitive), with only very minor performance difference. So even though it was my suggestion to change it, I'm leaning towards the current implementation :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655580976
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
Review comment:
+1
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSchemaHelperSuite.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Schema
+
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
+
+class AvroSchemaHelperSuite extends SQLTestUtils
+ with SharedSparkSession {
Review comment:
nit: move this line to the one above
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSchemaHelperSuite.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Schema
+
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
+
+class AvroSchemaHelperSuite extends SQLTestUtils
+ with SharedSparkSession {
+
+ test("ensure schema is a record") {
+ val jsonSchema = s"""
+ {
+ "namespace": "logical",
+ "type": "record",
+ "name": "test",
+ "fields": [
+ {"name": "date", "type": {"type": "int", "logicalType": "date"}}
+ ]
+ }
+ """
+
+ val avroSchema = Schema.parse(jsonSchema).getField("date").schema()
Review comment:
`Schema.parse` is Deprecated. Use Schema.Parser instead
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala
##########
@@ -31,7 +36,34 @@ package org.apache.spark.sql.execution.benchmark
* }}}
*/
object AvroWriteBenchmark extends DataSourceWriteBenchmark {
+ private def wideColumnsBenchmark: Unit = {
+ import spark.implicits._
+
+ withTempPath { dir =>
+ withTempTable("t1") {
+ val width = 1000
+ val values = 500000
+ val files = 20
+ val selectExpr = (1 to width).map(i => s"value as c$i")
+ // repartition to ensure we will write multiple files
+ val df = spark.range(values)
+ .map(_ => Random.nextInt).selectExpr(selectExpr: _*).repartition(files)
+ .persist(StorageLevel.DISK_ONLY)
+ // cache the data to ensure we are not benchmarking range or repartition
+ df.filter("(c1*c2) = 12").collect
Review comment:
what is this filtering for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866474658
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866085922
**[Test build #140148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140148/testReport)** for PR 32969 at commit [`ae61bdd`](https://github.com/apache/spark/commit/ae61bddd637a374759c7f9b54d24308cc4ea5d93).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866933088
Thanks for fixing this @bersprockets ! Much appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866471168
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864743352
Thanks for the ping. I will review this one today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655580976
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866452517
**[Test build #140168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140168/testReport)** for PR 32969 at commit [`d9e40d7`](https://github.com/apache/spark/commit/d9e40d747ffc26c16c7e40044388476db7767a7a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864683639
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44583/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-868156903
@bersprockets Thanks for reporting. Could you open a JIRA for the perf regression so that the community can follow up on it before the 3.2 release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866474639
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44694/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen edited a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
xkrogen edited a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866130603
@bersprockets -- @gengliangwang brought up the good point on #31490 that there is some interaction between these two PRs. I believe my changes in #31490 will also address the performance issue (I plan to run your benchmarks shortly), but also support matching fields by position (the main goal of #31490). Can you take a look there and see what you think?
Edit: Sorry, that's not quite right. The current code in #31490 is still doing N^2 matching. But the adjustment to the structure, where all of the field matching is done in a single method call instead of each method call matching one field, makes it easier to implement your changes IMO -- we can just construct the map within the new `matchCatalystAvroSchema` call. Here's the diff to implement your changes on top of #31490:
```
@@ -228,12 +227,15 @@ private[sql] object AvroUtils extends Logging {
throw new IncompatibleSchemaException(
s"Attempted to treat ${toFieldStr(avroPath)} as RECORD, but it was: ${avroSchema.getType}")
}
- val avroFields = avroSchema.getFields.asScala
+ val avroFieldMap = avroSchema.getFields.asScala.groupBy(_.name.toLowerCase(Locale.ROOT))
+ val avroFields = avroSchema.getFields.asScala.map(Some(_)).toArray
val catalystMatches = catalystType.zipWithIndex.map { case (catalystField, catalystPos) =>
val avroField = if (positionalFieldMatch) {
- avroFields.drop(catalystPos).headOption
+ avroFields.applyOrElse(catalystPos, _ => None)
} else {
- avroFields.filter(f => SQLConf.get.resolver(f.name(), catalystField.name)).toSeq match {
+ avroFieldMap
+ .getOrElse(catalystField.name.toLowerCase(Locale.ROOT), Nil)
+ .filter(f => SQLConf.get.resolver(f.name(), catalystField.name)).toSeq match {
case Seq(avroField) => Some(avroField)
case Seq() => None
case matches => throw new IncompatibleSchemaException("Searching for " +
@@ -244,8 +246,8 @@ private[sql] object AvroUtils extends Logging {
}
(Some(catalystField), avroField)
}
- val extraAvroFields = (avroFields.toSet -- catalystMatches.flatMap(_._2))
- .map(avroField => (None, Some(avroField)))
+ val extraAvroFields = (avroFields.toSet -- catalystMatches.flatMap(_._2).map(Some(_)))
+ .map(avroField => (None, avroField))
catalystMatches ++ extraAvroFields
}
```
I'm sure we can also implement #31490 on these changes -- just trying to think about what the end result will look like and minimize churn along the way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-865238182
@bersprockets Thanks a lot for fixing the perf regression!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866452517
**[Test build #140168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140168/testReport)** for PR 32969 at commit [`d9e40d7`](https://github.com/apache/spark/commit/d9e40d747ffc26c16c7e40044388476db7767a7a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864351056
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44537/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864345026
**[Test build #140011 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140011/testReport)** for PR 32969 at commit [`298a867`](https://github.com/apache/spark/commit/298a867750027c3331c096b3b55968ee70fb9c84).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r656362330
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSchemaHelperSuite.scala
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Schema
+
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
+
+class AvroSchemaHelperSuite extends SQLTestUtils with SharedSparkSession {
+
+ test("ensure schema is a record") {
+ val jsonSchema = s"""
+ {
+ "namespace": "logical",
+ "type": "record",
+ "name": "test",
+ "fields": [
+ {"name": "date", "type": {"type": "int", "logicalType": "date"}}
+ ]
+ }
+ """
+
+ val avroSchema = (new Schema.Parser()).parse(jsonSchema).getField("date").schema()
Review comment:
This seems a little overkill, maybe just something like:
```scala
val avroSchema = SchemaBuilder.builder().intType()
```
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSchemaHelperSuite.scala
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Schema
+
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
+
+class AvroSchemaHelperSuite extends SQLTestUtils with SharedSparkSession {
+
+ test("ensure schema is a record") {
+ val jsonSchema = s"""
+ {
+ "namespace": "logical",
+ "type": "record",
+ "name": "test",
+ "fields": [
+ {"name": "date", "type": {"type": "int", "logicalType": "date"}}
+ ]
+ }
+ """
+
+ val avroSchema = (new Schema.Parser()).parse(jsonSchema).getField("date").schema()
+ val msg = intercept[IncompatibleSchemaException] {
+ new AvroUtils.AvroSchemaHelper(avroSchema, Seq(""))
+ }.getMessage
+ assert(msg.contains("Attempting to treat int as a RECORD"))
+ }
+
+ test("handle mixed case field names") {
+ val catalystSchema = StructType(
+ StructField("a", IntegerType) ::
+ StructField("b", IntegerType) ::
+ StructField("A", IntegerType) ::
+ Nil
+ )
+
+ val avroSchema = SchemaConverters.toAvroType(catalystSchema)
+ val helper = new AvroUtils.AvroSchemaHelper(avroSchema, Seq(""))
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "true") {
+ assert(helper.getFieldByName("A").get.name() == "A")
+ assert(helper.getFieldByName("a").get.name() == "a")
+ assert(helper.getFieldByName("b").get.name() == "b")
+ assert(helper.getFieldByName("B").isEmpty)
+ }
+
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+ val msg1 = intercept[IncompatibleSchemaException] {
+ helper.getFieldByName("A")
+ }.getMessage
+ assert(msg1.contains("Searching for 'A' in Avro schema"))
+
+ val msg2 = intercept[IncompatibleSchemaException] {
+ helper.getFieldByName("a")
+ }.getMessage
+ assert(msg2.contains("Searching for 'a' in Avro schema"))
Review comment:
Consolidate?
```scala
Seq("a", "A").foreach { fieldName =>
withClue(s"looking for field name: $fieldName") {
val msg = intercept[IncompatibleSchemaException] {
helper.getFieldByName(fieldName)
}.getMessage
assert(msg.contains(s"Searching for '$fieldName' in Avro schema"))
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864473798
**[Test build #140032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140032/testReport)** for PR 32969 at commit [`7e0bb71`](https://github.com/apache/spark/commit/7e0bb71ee86f5a5c384e3eb8a57dea0b786684af).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866462249
**[Test build #140168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140168/testReport)** for PR 32969 at commit [`d9e40d7`](https://github.com/apache/spark/commit/d9e40d747ffc26c16c7e40044388476db7767a7a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang edited a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang edited a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866893947
Merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r656379707
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
+
+ /**
+ * Extract a single field from the contained avro schema which has the desired field name,
+ * performing the matching with proper case sensitivity according to SQLConf.resolver.
+ *
+ * @param name The name of the field to search for.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ def getFieldByName(name: String): Option[Schema.Field] = {
+
+ // get candidates, ignoring case of field name
+ val candidates = schemaMap.get(name.toLowerCase(Locale.ROOT))
+ .getOrElse(Seq.empty[Schema.Field])
+
+ // search candidates, taking into account case sensitivity settings
+ candidates.filter(f => SQLConf.get.resolver(f.name(), name)) match {
+ case Seq(avroField) => Some(avroField)
Review comment:
The current approach LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866138195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866138195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864479784
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864341696
**[Test build #140011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140011/testReport)** for PR 32969 at commit [`298a867`](https://github.com/apache/spark/commit/298a867750027c3331c096b3b55968ee70fb9c84).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864658300
**[Test build #140055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140055/testReport)** for PR 32969 at commit [`9e79846`](https://github.com/apache/spark/commit/9e79846009b4c5ca3a227054e031f8a98b65de87).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866441197
>minimize churn along the way.
I see your point. I will leave it up to you guys. If it gets fixed, here or elsewhere, that's good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866473297
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140168/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-865517679
Looks pretty good to me too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864665689
**[Test build #140055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140055/testReport)** for PR 32969 at commit [`9e79846`](https://github.com/apache/spark/commit/9e79846009b4c5ca3a227054e031f8a98b65de87).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864475047
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864341696
**[Test build #140011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140011/testReport)** for PR 32969 at commit [`298a867`](https://github.com/apache/spark/commit/298a867750027c3331c096b3b55968ee70fb9c84).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655588454
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSchemaHelperSuite.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Schema
+
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
+
+class AvroSchemaHelperSuite extends SQLTestUtils
+ with SharedSparkSession {
Review comment:
nit: move this line to the one above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655590235
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSchemaHelperSuite.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Schema
+
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
+import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
+
+class AvroSchemaHelperSuite extends SQLTestUtils
+ with SharedSparkSession {
+
+ test("ensure schema is a record") {
+ val jsonSchema = s"""
+ {
+ "namespace": "logical",
+ "type": "record",
+ "name": "test",
+ "fields": [
+ {"name": "date", "type": {"type": "int", "logicalType": "date"}}
+ ]
+ }
+ """
+
+ val avroSchema = Schema.parse(jsonSchema).getField("date").schema()
Review comment:
`Schema.parse` is Deprecated. Use Schema.Parser instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866893947
Merging to master and branch-3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864471563
**[Test build #140032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140032/testReport)** for PR 32969 at commit [`7e0bb71`](https://github.com/apache/spark/commit/7e0bb71ee86f5a5c384e3eb8a57dea0b786684af).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655548950
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
Review comment:
minor nit: `private[this] val`
also maybe `fieldMap` instead of `schemaMap` ?
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
##########
@@ -250,11 +250,12 @@ private[sql] class AvroSerializer(
s"Avro $avroPathStr schema length (${avroFields.size}) doesn't match " +
s"SQL ${toFieldStr(catalystPath)} schema length (${catalystStruct.length})")
}
+ val avroSchemaHelper = new AvroUtils.AvroSchemaHelper(avroStruct, avroPath)
val (avroIndices: Array[Int], fieldConverters: Array[Converter]) =
catalystStruct.map { catalystField =>
- val avroField = AvroUtils
- .getAvroFieldByName(avroStruct, catalystField.name, avroPath) match {
+ val avroField = avroSchemaHelper
+ .getFieldByName(catalystField.name) match {
Review comment:
minor nit: you can combine these two lines back into 1 now that the method call is shorter
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
Review comment:
can you keep the Scaladoc for `avroSchema` and `avroPath` from the previous version?
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
+
+ /**
+ * Extract a single field from the contained avro schema which has the desired field name,
+ * performing the matching with proper case sensitivity according to SQLConf.resolver.
+ *
+ * @param name The name of the field to search for.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ def getFieldByName(name: String): Option[Schema.Field] = {
+
+ // get candidates, ignoring case of field name
+ val candidates = schemaMap.get(name.toLowerCase(Locale.ROOT))
+ .getOrElse(Seq.empty[Schema.Field])
+
+ // search candidates, taking into account case sensitivity settings
+ candidates.filter(f => SQLConf.get.resolver(f.name(), name)) match {
+ case Seq(avroField) => Some(avroField)
Review comment:
Can you provide any commentary on why you chose to always lowercase and then resolve later based on case sensitivity, vs. storing the map keys as case insensitive or sensitive based on the SQL conf? Is it so that we can make use of `SQLConf#resolver` to perform the resolution instead of checking the value of the case sensitive conf ourselves?
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
Review comment:
Formatting here looks a little odd to me, maybe we can do:
```
avroSchema.getFields.asScala
.groupBy(_.name.toLowerCase(Locale.ROOT))
.mapValues(_.toSeq) // toSeq needed for scala 2.13
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866473297
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655802350
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
+
+ /**
+ * Extract a single field from the contained avro schema which has the desired field name,
+ * performing the matching with proper case sensitivity according to SQLConf.resolver.
+ *
+ * @param name The name of the field to search for.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ def getFieldByName(name: String): Option[Schema.Field] = {
+
+ // get candidates, ignoring case of field name
+ val candidates = schemaMap.get(name.toLowerCase(Locale.ROOT))
+ .getOrElse(Seq.empty[Schema.Field])
+
+ // search candidates, taking into account case sensitivity settings
+ candidates.filter(f => SQLConf.get.resolver(f.name(), name)) match {
+ case Seq(avroField) => Some(avroField)
Review comment:
>Is it so that we can make use of SQLConf#resolver to perform the resolution
Well, that was certainly one reason (so I don't have to decide the resolver, in case that decision logic changes).
But I also wanted to keep the code similar to the original implementation (i.e., duplicates are detected at lookup time, etc.).
I suppose my version wouldn't help much in the case of a wide table where all fields map to the same lower case name (e.g., a 10 character field name that just repeats with many different permutations of case). Hopefully that is an extremely unusual situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655794745
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala
##########
@@ -31,7 +36,34 @@ package org.apache.spark.sql.execution.benchmark
* }}}
*/
object AvroWriteBenchmark extends DataSourceWriteBenchmark {
+ private def wideColumnsBenchmark: Unit = {
+ import spark.implicits._
+
+ withTempPath { dir =>
+ withTempTable("t1") {
+ val width = 1000
+ val values = 500000
+ val files = 20
+ val selectExpr = (1 to width).map(i => s"value as c$i")
+ // repartition to ensure we will write multiple files
+ val df = spark.range(values)
+ .map(_ => Random.nextInt).selectExpr(selectExpr: _*).repartition(files)
+ .persist(StorageLevel.DISK_ONLY)
+ // cache the data to ensure we are not benchmarking range or repartition
+ df.filter("(c1*c2) = 12").collect
Review comment:
I think I can replace this with `df.noop()`. I will check.
I was using `collect` to force the evaluation (and caching) of df. But I didn't want to actually collect 500K wide rows, so I filtered away all of df's rows.
The whacky filter expression `(c1*c2) = 12` was to ensure no push-down to the file format's libraries (so that df actually reads all rows). Not really needed in the case of Avro.
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
+
+ /**
+ * Extract a single field from the contained avro schema which has the desired field name,
+ * performing the matching with proper case sensitivity according to SQLConf.resolver.
+ *
+ * @param name The name of the field to search for.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ def getFieldByName(name: String): Option[Schema.Field] = {
+
+ // get candidates, ignoring case of field name
+ val candidates = schemaMap.get(name.toLowerCase(Locale.ROOT))
+ .getOrElse(Seq.empty[Schema.Field])
+
+ // search candidates, taking into account case sensitivity settings
+ candidates.filter(f => SQLConf.get.resolver(f.name(), name)) match {
+ case Seq(avroField) => Some(avroField)
Review comment:
>Is it so that we can make use of SQLConf#resolver to perform the resolution
Well, that was certainly one reason (so I don't have to decide the resolver, in case that decision logic changes).
But I also wanted to keep the code similar to the original implementation (i.e., duplicates are detected at lookup time, etc.).
I suppose my version wouldn't help much in the case of a wide table where all fields map to the same lower case name (e.g., a 10 character field name that just repeats with many different permutations of case). Hopefully that is an extremely unusual situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864658300
**[Test build #140055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140055/testReport)** for PR 32969 at commit [`9e79846`](https://github.com/apache/spark/commit/9e79846009b4c5ca3a227054e031f8a98b65de87).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864474930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140032/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-869062066
@gengliangwang It's no longer reproducible. branch-3.1 (at the HEAD and even at older commits) runs similar to master. I won't file a Jira at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864347257
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140011/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864347257
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140011/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864676140
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140055/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866085922
**[Test build #140148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140148/testReport)** for PR 32969 at commit [`ae61bdd`](https://github.com/apache/spark/commit/ae61bddd637a374759c7f9b54d24308cc4ea5d93).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655548950
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
Review comment:
minor nit: `private[this] val`
also maybe `fieldMap` instead of `schemaMap` ?
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
##########
@@ -250,11 +250,12 @@ private[sql] class AvroSerializer(
s"Avro $avroPathStr schema length (${avroFields.size}) doesn't match " +
s"SQL ${toFieldStr(catalystPath)} schema length (${catalystStruct.length})")
}
+ val avroSchemaHelper = new AvroUtils.AvroSchemaHelper(avroStruct, avroPath)
val (avroIndices: Array[Int], fieldConverters: Array[Converter]) =
catalystStruct.map { catalystField =>
- val avroField = AvroUtils
- .getAvroFieldByName(avroStruct, catalystField.name, avroPath) match {
+ val avroField = avroSchemaHelper
+ .getFieldByName(catalystField.name) match {
Review comment:
minor nit: you can combine these two lines back into 1 now that the method call is shorter
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
Review comment:
can you keep the Scaladoc for `avroSchema` and `avroPath` from the previous version?
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
+
+ /**
+ * Extract a single field from the contained avro schema which has the desired field name,
+ * performing the matching with proper case sensitivity according to SQLConf.resolver.
+ *
+ * @param name The name of the field to search for.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ def getFieldByName(name: String): Option[Schema.Field] = {
+
+ // get candidates, ignoring case of field name
+ val candidates = schemaMap.get(name.toLowerCase(Locale.ROOT))
+ .getOrElse(Seq.empty[Schema.Field])
+
+ // search candidates, taking into account case sensitivity settings
+ candidates.filter(f => SQLConf.get.resolver(f.name(), name)) match {
+ case Seq(avroField) => Some(avroField)
Review comment:
Can you provide any commentary on why you chose to always lowercase and then resolve later based on case sensitivity, vs. storing the map keys as case insensitive or sensitive based on the SQL conf? Is it so that we can make use of `SQLConf#resolver` to perform the resolution instead of checking the value of the case sensitive conf ourselves?
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
Review comment:
Formatting here looks a little odd to me, maybe we can do:
```
avroSchema.getFields.asScala
.groupBy(_.name.toLowerCase(Locale.ROOT))
.mapValues(_.toSeq) // toSeq needed for scala 2.13
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864352442
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44537/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r654733861
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala
##########
@@ -31,7 +36,34 @@ package org.apache.spark.sql.execution.benchmark
* }}}
*/
object AvroWriteBenchmark extends DataSourceWriteBenchmark {
+ private def wideColumnsBenchmark: Unit = {
+ import spark.implicits._
+
+ withTempPath { dir =>
+ withTempTable("t1") {
+ val width = 1000
+ val values = 500000
+ val files = 20
+ val selectExpr = (1 to width).map(i => s"value as c$i")
+ // repartition to ensure we will write multiple files
+ val df = spark.range(values)
+ .map(_ => Random.nextInt).selectExpr(selectExpr: _*).repartition(files)
+ .persist(StorageLevel.DISK_ONLY)
+ // cache the data to ensure we are not benchmarking range or repartition
+ df.filter("(c1*c2) = 12").collect
+ df.createOrReplaceTempView("t1")
+ val benchmark = new Benchmark(s"Write wide rows into $files files", values)
+ benchmark.addCase("Write wide rows") { _ =>
+ spark.sql("SELECT * FROM t1").
+ write.format("avro").save(s"${dir.getCanonicalPath}/${Random.nextLong.abs}")
+ }
+ benchmark.run()
Review comment:
This is not quite working. Results for this benchmark get printed in stdout, but they don't show up in AvroWriteBenchmark-results.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864695678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44583/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r656380714
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala
##########
@@ -31,7 +36,34 @@ package org.apache.spark.sql.execution.benchmark
* }}}
*/
object AvroWriteBenchmark extends DataSourceWriteBenchmark {
+ private def wideColumnsBenchmark: Unit = {
+ import spark.implicits._
+
+ withTempPath { dir =>
+ withTempTable("t1") {
+ val width = 1000
+ val values = 500000
+ val files = 20
+ val selectExpr = (1 to width).map(i => s"value as c$i")
+ // repartition to ensure we will write multiple files
+ val df = spark.range(values)
+ .map(_ => Random.nextInt).selectExpr(selectExpr: _*).repartition(files)
+ .persist(StorageLevel.DISK_ONLY)
+ // cache the data to ensure we are not benchmarking range or repartition
+ df.noop()
Review comment:
shall we update the benchmark result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866130603
@bersprockets -- @gengliangwang brought up the good point on #31490 that there is some interaction between these two PRs. I believe my changes in #31490 will also address the performance issue (I plan to run your benchmarks shortly), but also support matching fields by position (the main goal of #31490). Can you take a look there and see what you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r656329916
##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -202,34 +203,40 @@ private[sql] object AvroUtils extends Logging {
}
}
- /**
- * Extract a single field from `avroSchema` which has the desired field name,
- * performing the matching with proper case sensitivity according to [[SQLConf.resolver]].
- *
- * @param avroSchema The schema in which to search for the field. Must be of type RECORD.
- * @param name The name of the field to search for.
- * @param avroPath The seq of parent field names leading to `avroSchema`.
- * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
- * @throws IncompatibleSchemaException if `avroSchema` is not a RECORD or contains multiple
- * fields matching `name` (i.e., case-insensitive matching
- * is used and `avroSchema` has two or more fields that have
- * the same name with difference case).
- */
- private[avro] def getAvroFieldByName(
- avroSchema: Schema,
- name: String,
- avroPath: Seq[String]): Option[Schema.Field] = {
+ class AvroSchemaHelper(avroSchema: Schema, avroPath: Seq[String]) {
if (avroSchema.getType != Schema.Type.RECORD) {
throw new IncompatibleSchemaException(
s"Attempting to treat ${avroSchema.getName} as a RECORD, but it was: ${avroSchema.getType}")
}
- avroSchema.getFields.asScala.filter(f => SQLConf.get.resolver(f.name(), name)).toSeq match {
- case Seq(avroField) => Some(avroField)
- case Seq() => None
- case matches => throw new IncompatibleSchemaException(s"Searching for '$name' in Avro " +
+
+ val schemaMap = avroSchema.getFields.asScala.groupBy { f =>
+ f.name.toLowerCase(Locale.ROOT)
+ }.map { case (k, v) =>
+ (k, v.toSeq) // needed for scala 2.13
+ }
+
+ /**
+ * Extract a single field from the contained avro schema which has the desired field name,
+ * performing the matching with proper case sensitivity according to SQLConf.resolver.
+ *
+ * @param name The name of the field to search for.
+ * @return `Some(match)` if a matching Avro field is found, otherwise `None`.
+ */
+ def getFieldByName(name: String): Option[Schema.Field] = {
+
+ // get candidates, ignoring case of field name
+ val candidates = schemaMap.get(name.toLowerCase(Locale.ROOT))
+ .getOrElse(Seq.empty[Schema.Field])
+
+ // search candidates, taking into account case sensitivity settings
+ candidates.filter(f => SQLConf.get.resolver(f.name(), name)) match {
+ case Seq(avroField) => Some(avroField)
Review comment:
@xkrogen @gengliangwang What do you think (referring to xkrogen's comment above)? Should we instead build a map of `String -> Schema.Field` rather than `String -> Seq[Schema.Field]`, using the value of spark.sql.caseSensitive to determine whether the keys are mixed case or single case (and checking for duplicates, in the case insensitive mode, at Map build time, rather than lookup time)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864477253
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864676140
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140055/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866897972
@bersprockets There are some code conflicts when merging to branch-3.1. Could you open a cherry-pick PR for branch-3.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864474930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140032/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864695678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44583/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864479784
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864673704
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44583/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866939304
@xkrogen @gengliangwang Thanks for the help! I will open the PR to branch-3.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-865517679
Looks pretty good to me too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets edited a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets edited a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-867829614
@gengliangwang @xkrogen Just something someone might want to check: In preparing the PR for branch-3.1, I noticed that the Avro benchmarks for branch-3.1 run about 15-25% faster, across the board, than the Avro benchmarks for master. I didn't check to see if that's also true for other benchmarks (Parquet, ORC, etc.).
I've been using the same AWS VM to run these benchmarks.
Note: this is a general thing between branch-3.1 and master. It's not related to this fix.
Edit: After updating my master working copy, the difference is less dramatic: 2-16% in 3.1's favor, with all the "Sum of string length" benchmarks running much faster on master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866115044
**[Test build #140148 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140148/testReport)** for PR 32969 at commit [`ae61bdd`](https://github.com/apache/spark/commit/ae61bddd637a374759c7f9b54d24308cc4ea5d93).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r654733861
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala
##########
@@ -31,7 +36,34 @@ package org.apache.spark.sql.execution.benchmark
* }}}
*/
object AvroWriteBenchmark extends DataSourceWriteBenchmark {
+ private def wideColumnsBenchmark: Unit = {
+ import spark.implicits._
+
+ withTempPath { dir =>
+ withTempTable("t1") {
+ val width = 1000
+ val values = 500000
+ val files = 20
+ val selectExpr = (1 to width).map(i => s"value as c$i")
+ // repartition to ensure we will write multiple files
+ val df = spark.range(values)
+ .map(_ => Random.nextInt).selectExpr(selectExpr: _*).repartition(files)
+ .persist(StorageLevel.DISK_ONLY)
+ // cache the data to ensure we are not benchmarking range or repartition
+ df.filter("(c1*c2) = 12").collect
+ df.createOrReplaceTempView("t1")
+ val benchmark = new Benchmark(s"Write wide rows into $files files", values)
+ benchmark.addCase("Write wide rows") { _ =>
+ spark.sql("SELECT * FROM t1").
+ write.format("avro").save(s"${dir.getCanonicalPath}/${Random.nextLong.abs}")
+ }
+ benchmark.run()
Review comment:
This is not quite working. Results for this benchmark get printed in stdout, but they don't show up in AvroWriteBenchmark-results.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866133660
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44674/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-865238182
@bersprockets Thanks a lot for fixing the perf regression!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864471563
**[Test build #140032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140032/testReport)** for PR 32969 at commit [`7e0bb71`](https://github.com/apache/spark/commit/7e0bb71ee86f5a5c384e3eb8a57dea0b786684af).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864491618
cc @gengliangwang FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655597239
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala
##########
@@ -31,7 +36,34 @@ package org.apache.spark.sql.execution.benchmark
* }}}
*/
object AvroWriteBenchmark extends DataSourceWriteBenchmark {
+ private def wideColumnsBenchmark: Unit = {
+ import spark.implicits._
+
+ withTempPath { dir =>
+ withTempTable("t1") {
+ val width = 1000
+ val values = 500000
+ val files = 20
+ val selectExpr = (1 to width).map(i => s"value as c$i")
+ // repartition to ensure we will write multiple files
+ val df = spark.range(values)
+ .map(_ => Random.nextInt).selectExpr(selectExpr: _*).repartition(files)
+ .persist(StorageLevel.DISK_ONLY)
+ // cache the data to ensure we are not benchmarking range or repartition
+ df.filter("(c1*c2) = 12").collect
Review comment:
what is this filtering for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-867829614
@gengliangwang @xkrogen Just something someone might want to check: In preparing the PR for branch-3.1, I noticed that the Avro benchmarks for branch-3.1 run about 15-25% faster, across the board, than the Avro benchmarks for master. I didn't check to see if that's also true for other benchmarks (Parquet, ORC, etc.).
I've been using the same AWS VM to run these benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a change in pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets commented on a change in pull request #32969:
URL: https://github.com/apache/spark/pull/32969#discussion_r655794745
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala
##########
@@ -31,7 +36,34 @@ package org.apache.spark.sql.execution.benchmark
* }}}
*/
object AvroWriteBenchmark extends DataSourceWriteBenchmark {
+ private def wideColumnsBenchmark: Unit = {
+ import spark.implicits._
+
+ withTempPath { dir =>
+ withTempTable("t1") {
+ val width = 1000
+ val values = 500000
+ val files = 20
+ val selectExpr = (1 to width).map(i => s"value as c$i")
+ // repartition to ensure we will write multiple files
+ val df = spark.range(values)
+ .map(_ => Random.nextInt).selectExpr(selectExpr: _*).repartition(files)
+ .persist(StorageLevel.DISK_ONLY)
+ // cache the data to ensure we are not benchmarking range or repartition
+ df.filter("(c1*c2) = 12").collect
Review comment:
I think I can replace this with `df.noop()`. I will check.
I was using `collect` to force the evaluation (and caching) of df. But I didn't want to actually collect 500K wide rows, so I filtered away all of df's rows.
The whacky filter expression `(c1*c2) = 12` was to ensure no push-down to the file format's libraries (so that df actually reads all rows). Not really needed in the case of Avro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen edited a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
xkrogen edited a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-866130603
@bersprockets -- @gengliangwang brought up the good point on #31490 that there is some interaction between these two PRs. I believe my changes in #31490 will also address the performance issue (I plan to run your benchmarks shortly), but also support matching fields by position (the main goal of #31490). Can you take a look there and see what you think?
Edit: Sorry, that's not quite right. The current code in #31490 is still doing N^2 matching. But the adjustment to the structure, where all of the field matching is done in a single method call instead of each method call matching one field, makes it easier to implement your changes IMO -- we can just construct the map within the new `matchCatalystAvroSchema` call. Here's the diff to implement your changes on top of #31490:
```
@@ -228,12 +227,15 @@ private[sql] object AvroUtils extends Logging {
throw new IncompatibleSchemaException(
s"Attempted to treat ${toFieldStr(avroPath)} as RECORD, but it was: ${avroSchema.getType}")
}
- val avroFields = avroSchema.getFields.asScala
+ val avroFieldMap = avroSchema.getFields.asScala.groupBy(_.name.toLowerCase(Locale.ROOT))
+ val avroFields = avroSchema.getFields.asScala.map(Some(_)).toArray
val catalystMatches = catalystType.zipWithIndex.map { case (catalystField, catalystPos) =>
val avroField = if (positionalFieldMatch) {
- avroFields.drop(catalystPos).headOption
+ avroFields.applyOrElse(catalystPos, _ => None)
} else {
- avroFields.filter(f => SQLConf.get.resolver(f.name(), catalystField.name)).toSeq match {
+ avroFieldMap
+ .getOrElse(catalystField.name.toLowerCase(Locale.ROOT), Nil)
+ .filter(f => SQLConf.get.resolver(f.name(), catalystField.name)).toSeq match {
case Seq(avroField) => Some(avroField)
case Seq() => None
case matches => throw new IncompatibleSchemaException("Searching for " +
@@ -244,8 +246,8 @@ private[sql] object AvroUtils extends Logging {
}
(Some(catalystField), avroField)
}
- val extraAvroFields = (avroFields.toSet -- catalystMatches.flatMap(_._2))
- .map(avroField => (None, Some(avroField)))
+ val extraAvroFields = (avroFields.toSet -- catalystMatches.flatMap(_._2).map(Some(_)))
+ .map(avroField => (None, avroField))
catalystMatches ++ extraAvroFields
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets edited a comment on pull request #32969: [SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
bersprockets edited a comment on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-867829614
@gengliangwang @xkrogen Just something someone might want to check: In preparing the PR for branch-3.1, I noticed that the Avro benchmarks for branch-3.1 run about 15-25% faster, across the board, than the Avro benchmarks for master. I didn't check to see if that's also true for other benchmarks (Parquet, ORC, etc.).
I've been using the same AWS VM to run these benchmarks.
Note: this is a general thing between branch-3.1 and master. It's not related to this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864352442
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44537/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32969: [WIP][SPARK-35817][SQL] Restore performance of queries against wide Avro tables
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32969:
URL: https://github.com/apache/spark/pull/32969#issuecomment-864347830
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44537/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org