You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/01/09 17:53:43 UTC
[spark] branch master updated: [SPARK-41860][SQL] Make AvroScanBuilder and JsonScanBuilder case classes
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1f1c0016fad [SPARK-41860][SQL] Make AvroScanBuilder and JsonScanBuilder case classes
1f1c0016fad is described below
commit 1f1c0016fad234ba9d939a8269adc3059842687a
Author: Lorenzo Martini <lo...@gmail.com>
AuthorDate: Mon Jan 9 09:53:22 2023 -0800
[SPARK-41860][SQL] Make AvroScanBuilder and JsonScanBuilder case classes
### What changes were proposed in this pull request?
Make `AvroScanBuilder` and `JsonScanBuilder` case classes from normal classes.
All the other existing `ScanBuilder`s inheriting from `FileScanBuilder` are case classes which is very nice and useful. However the Json and the Avro ones aren't, which makes it quite inconvenient when working with those. I don't see any particular reason why these two would not be case classes as well given they are very similar to the other ones and very simple and stateless. Especially when `AvroScan` and `JsonScan` are also already case classes like the other scans.
### Why are the changes needed?
When working on using different possible `FileScanBuilder` is really problematic that these 2 ScanBuilders are not case classes, especially given all the others are. This is a very simple change that helps development greatly and allows for more flexibility / feature deelopment when working with ScanBuilders.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No need for testing as it's a simple change from `class` to `case class`
Closes #39366 from LorenzoMartini/patch-1.
Authored-by: Lorenzo Martini <lo...@gmail.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala | 2 +-
.../spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala b/connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala
index 08dcaa7d087..754c58e65b0 100644
--- a/connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala
+++ b/connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala
@@ -24,7 +24,7 @@ import org.apache.spark.sql.sources.Filter
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.util.CaseInsensitiveStringMap
-class AvroScanBuilder (
+case class AvroScanBuilder (
sparkSession: SparkSession,
fileIndex: PartitioningAwareFileIndex,
schema: StructType,
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala
index 788ccfac6e7..dcae6bd3fd0 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala
@@ -24,7 +24,7 @@ import org.apache.spark.sql.sources.Filter
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.util.CaseInsensitiveStringMap
-class JsonScanBuilder (
+case class JsonScanBuilder (
sparkSession: SparkSession,
fileIndex: PartitioningAwareFileIndex,
schema: StructType,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org