You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yyanyy (via GitHub)" <gi...@apache.org> on 2026/03/28 00:40:01 UTC

[PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

yyanyy opened a new pull request, #55070:
URL: https://github.com/apache/spark/pull/55070

   
   ### What changes were proposed in this pull request?
   
   `DataSourceV2ScanRelation` is a case class with 5 fields. Many pattern match sites only need the `scan` field or a subset of `(relation, scan, output)`, using wildcards for the rest. This couples every match site to the constructor arity, so adding or removing fields requires updating all of them.
   
   This PR introduces two extractors following the `ExtractV2Table` precedent (SPARK-53720):
   - `ExtractV2Scan`: returns `Scan`
   - `ExtractV2ScanRelation`: returns `(DataSourceV2Relation, Scan, Seq[AttributeReference])`
   
   Updated 14 pattern match sites across 10 files. Type-based matches and constructor calls are unchanged.
   
   ### Why are the changes needed?
   Code simplification. Similar to https://github.com/apache/spark/commit/6bae835ccbc8850ac5e2ab0225c6cd75921f06b4 
   
   ### Does this PR introduce _any_ user-facing change?
   no
   
   ### How was this patch tested?
   This PR relies on existing tests.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   Yes - Opus 4.6 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3029541352


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {

Review Comment:
   We should still think a bit more about the name here. It does extract a relation but it does this in a scan context and the name doesn't say anything about that. I agree `ExtractV2ScanRelation` is also somewhat not accurate as we are not extracting it. I wonder if things like `ExtractV2ScanInfo` would be better.
   
   Let me think. Name alternatives are welcome.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #55070:
URL: https://github.com/apache/spark/pull/55070#issuecomment-4180019423

   I support this idea as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "yyanyy (via GitHub)" <gi...@apache.org>.
yyanyy commented on PR #55070:
URL: https://github.com/apache/spark/pull/55070#issuecomment-4158690272

   Thank you both for the quick review! I have updated the PR per comment, please take a look again when you have a chance


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "yyanyy (via GitHub)" <gi...@apache.org>.
yyanyy commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3030478188


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {

Review Comment:
   I'm sure `ExtractV2ScanRelationInfo` has crossed some of your mind but it's probably too long?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3008193780


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -202,6 +202,32 @@ case class DataSourceV2ScanRelation(
   }
 }
 
+/**
+ * An extractor that returns the [[Scan]] from a [[DataSourceV2ScanRelation]].
+ *
+ * This is useful when a pattern match site only needs the scan and should not be

Review Comment:
   +1



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -202,6 +202,32 @@ case class DataSourceV2ScanRelation(
   }
 }
 
+/**
+ * An extractor that returns the [[Scan]] from a [[DataSourceV2ScanRelation]].
+ *
+ * This is useful when a pattern match site only needs the scan and should not be
+ * coupled to the full constructor signature of [[DataSourceV2ScanRelation]].
+ */
+object ExtractV2Scan {

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55070:
URL: https://github.com/apache/spark/pull/55070#issuecomment-4195660980

   the failed test is unrelated, thanks, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "yyanyy (via GitHub)" <gi...@apache.org>.
yyanyy commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3030463854


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {
+  def unapply(scanRelation: DataSourceV2ScanRelation)
+      : Option[(DataSourceV2Relation, Scan, Seq[AttributeReference])] =

Review Comment:
   3 of the 6 instances that requires `ExtractV2Relation` would need `output` but all usages are currently in `DataSourceV2Strategy` so I guess we can consider it either way... 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3004206786


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -202,6 +202,32 @@ case class DataSourceV2ScanRelation(
   }
 }
 
+/**
+ * An extractor that returns the [[Scan]] from a [[DataSourceV2ScanRelation]].
+ *
+ * This is useful when a pattern match site only needs the scan and should not be
+ * coupled to the full constructor signature of [[DataSourceV2ScanRelation]].
+ */
+object ExtractV2Scan {

Review Comment:
   nit: should we move it to the other Extractor for DSV2Relation below (like ExtractV2Table)



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -202,6 +202,32 @@ case class DataSourceV2ScanRelation(
   }
 }
 
+/**
+ * An extractor that returns the [[Scan]] from a [[DataSourceV2ScanRelation]].
+ *
+ * This is useful when a pattern match site only needs the scan and should not be

Review Comment:
   nit: should we remove this comment? (thinking, else why not put it on all the extractors?)



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -202,6 +202,32 @@ case class DataSourceV2ScanRelation(
   }
 }
 
+/**
+ * An extractor that returns the [[Scan]] from a [[DataSourceV2ScanRelation]].
+ *
+ * This is useful when a pattern match site only needs the scan and should not be
+ * coupled to the full constructor signature of [[DataSourceV2ScanRelation]].
+ */
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+/**
+ * An extractor that returns a 3-tuple of ([[DataSourceV2Relation]], [[Scan]],
+ * output attributes) from a [[DataSourceV2ScanRelation]].
+ *
+ * This is useful when a pattern match site needs the relation, scan, and output

Review Comment:
   same, should we remove?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3008206743


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -202,6 +202,32 @@ case class DataSourceV2ScanRelation(
   }
 }
 
+/**
+ * An extractor that returns the [[Scan]] from a [[DataSourceV2ScanRelation]].
+ *
+ * This is useful when a pattern match site only needs the scan and should not be
+ * coupled to the full constructor signature of [[DataSourceV2ScanRelation]].
+ */
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+/**
+ * An extractor that returns a 3-tuple of ([[DataSourceV2Relation]], [[Scan]],
+ * output attributes) from a [[DataSourceV2ScanRelation]].
+ *
+ * This is useful when a pattern match site needs the relation, scan, and output
+ * but should not be coupled to the full constructor signature of
+ * [[DataSourceV2ScanRelation]] (e.g., optional fields like keyGroupedPartitioning
+ * and ordering).
+ */
+object ExtractV2ScanRelation {
+  def unapply(scanRelation: DataSourceV2ScanRelation)
+      : Option[(DataSourceV2Relation, Scan, Seq[AttributeReference])] =

Review Comment:
   `ExtractV2Scan` returns v2 scan, this returns `DataSourceV2Relation`, shall we name it `ExtractV2Relation`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #55070: [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation
URL: https://github.com/apache/spark/pull/55070


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3029758352


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {

Review Comment:
   +1 to change the name to indicate it extract more than relation (sorry didnt reply earlier to https://github.com/apache/spark/pull/55070#discussion_r3008206743)  ExtractV2ScanComponents is another one, but maybe Info is better as its shorter



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3029872626


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {
+  def unapply(scanRelation: DataSourceV2ScanRelation)
+      : Option[(DataSourceV2Relation, Scan, Seq[AttributeReference])] =

Review Comment:
   Is it common that we need output as well? Technically, you could do like this:
   
   ```
   r @ ExtractV2Relation(_, scan: LocalScan) => r.output
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "anton5798 (via GitHub)" <gi...@apache.org>.
anton5798 commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3029976313


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {

Review Comment:
   nit: do we have to include `Extract` in the name? might be cleaner w/o it (it's already assumed it is an extractor..). `V2RelationAndScan` / `V2Scan` / `V2ScanContext`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3030306286


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {

Review Comment:
   @anton5798, I had a similar question when we did extractors for DataSourceV2Relation but the consensus was that it is beneficial to have `ExtractXXX` explicitly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "yyanyy (via GitHub)" <gi...@apache.org>.
yyanyy commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3030500464


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {

Review Comment:
   I think I like `ExtractV2ScanInfo` the most and still include output, let me update the PR with this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3029884724


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {

Review Comment:
   If we end up not extracting output, it could be something like: `ExtractV2RelationAndScan`.
   
   The key ask: the name has to indicate it applies to scanning.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3029887080


##########
sql/core/src/test/scala/org/apache/spark/sql/collation/CollatedFilterPushDownToParquetSuite.scala:
##########
@@ -258,7 +258,7 @@ class CollatedFilterPushDownToParquetV2Suite extends CollatedFilterPushDownToPar
   override def getPushedDownFilters(query: DataFrame): Seq[Filter] = {
     query.queryExecution.optimizedPlan.collectFirst {
       case PhysicalOperation(_, _,
-          DataSourceV2ScanRelation(_, scan: ParquetScan, _, _, _)) =>
+          ExtractV2Scan(scan: ParquetScan)) =>

Review Comment:
   Can this now be moved to the same line?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56273][SQL] Simplify extracting fields from DataSourceV2ScanRelation [spark]

Posted by "anton5798 (via GitHub)" <gi...@apache.org>.
anton5798 commented on code in PR #55070:
URL: https://github.com/apache/spark/pull/55070#discussion_r3029976313


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -285,6 +285,17 @@ object ExtractV2CatalogAndIdentifier {
   }
 }
 
+object ExtractV2Scan {
+  def unapply(scanRelation: DataSourceV2ScanRelation): Option[Scan] =
+    Some(scanRelation.scan)
+}
+
+object ExtractV2Relation {

Review Comment:
   nit: do we have to include `Extract` in the name? might be cleaner w/o it (it's already assumed it is an extractor..). `V2RelationAndScan` / `V2Relation` / `V2ScanContext`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org