You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Kimahriman (via GitHub)" <gi...@apache.org> on 2024/09/05 14:50:29 UTC

[PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Kimahriman opened a new pull request, #915:
URL: https://github.com/apache/datafusion-comet/pull/915

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #912 
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   Fixes CometScanExec running on Spark 3.5.2+
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   Changes the verbose string output for CometScanExec. Instead of delegating to the underlying DataSourceScanExec, just copied the implementation over. This is in line with other Comet operators that implement their own verbose string, and makes more sense in the formatted explain as the operator names line up.
   
   Before:
   ```
   == Physical Plan ==
   * ColumnarToRow (2)
   +- CometScan parquet  (1)
   
   
   (1) Scan parquet 
   Output [20]: [_1#146, _2#147, _3#148, _4#149, _5#150L, _6#151, _7#152, _8#153, _9#154, _10#155, _11#156L, _12#157, _13#158, _14#159, _15#160, _16#161, _17#162, _18#163, _19#164, _20#165]
   Batched: true
   Location: InMemoryFileIndex [file:.../arrow-datafusion-comet/spark/target/tmp/spark-a0b56a3c-caa2-4631-ad15-ec83e8522948/test.parquet]
   ReadSchema: struct<_1:boolean,_2:tinyint,_3:smallint,_4:int,_5:bigint,_6:float,_7:double,_8:string,_9:smallint,_10:int,_11:bigint,_12:decimal(20,0),_13:string,_14:binary,_15:decimal(5,2),_16:decimal(18,10),_17:decimal(38,37),_18:timestamp,_19:timestamp,_20:date>
   
   (2) ColumnarToRow [codegen id : 1]
   Input [20]: [_1#146, _2#147, _3#148, _4#149, _5#150L, _6#151, _7#152, _8#153, _9#154, _10#155, _11#156L, _12#157, _13#158, _14#159, _15#160, _16#161, _17#162, _18#163, _19#164, _20#165]
   ```
   
   After:
   ```
   == Physical Plan ==
   * ColumnarToRow (2)
   +- CometScan parquet  (1)
   
   
   (1) CometScan parquet 
   Output [20]: [_1#146, _2#147, _3#148, _4#149, _5#150L, _6#151, _7#152, _8#153, _9#154, _10#155, _11#156L, _12#157, _13#158, _14#159, _15#160, _16#161, _17#162, _18#163, _19#164, _20#165]
   Batched: true
   Location: InMemoryFileIndex [file:.../arrow-datafusion-comet/spark/target/tmp/spark-36667e9a-75e0-4f90-8442-5f47ecd1cf4b/test.parquet]
   ReadSchema: struct<_1:boolean,_2:tinyint,_3:smallint,_4:int,_5:bigint,_6:float,_7:double,_8:string,_9:smallint,_10:int,_11:bigint,_12:decimal(20,0),_13:string,_14:binary,_15:decimal(5,2),_16:decimal(18,10),_17:decimal(38,37),_18:timestamp,_19:timestamp,_20:date>
   
   (2) ColumnarToRow [codegen id : 1]
   Input [20]: [_1#146, _2#147, _3#148, _4#149, _5#150L, _6#151, _7#152, _8#153, _9#154, _10#155, _11#156L, _12#157, _13#158, _14#159, _15#160, _16#161, _17#162, _18#163, _19#164, _20#165]
   ```
   
   ## How are these changes tested?
   Manually verified by building with Spark 3.5.2 `./mvnw clean package -Pspark-3.5 -Dspark.version=3.5.2 -DskipTests`.
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "Kimahriman (via GitHub)" <gi...@apache.org>.
Kimahriman commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2364344001

   > @Kimahriman would you be able to rebase this PR so that we can merge it?
   
   Oof 95 conflicts I would have to manually resolve, let me just regenerate these plans all again tonight


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2370815876

   > Done, I think CI failures are unrelated, looks like failures downloading dependencies in Hive tests
   
   Thanks. I am re-running the failed jobs now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2344878590

   And perhaps we can try to upgrade the Comet dependency to Spark 3.5.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2344868523

   Is the change in the `metadata`? If so, should we fix the metadata instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "Kimahriman (via GitHub)" <gi...@apache.org>.
Kimahriman commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2364453012

   Ok wasn't too bad to find/replace fix the issues again, we'll see if I messed anything up in the CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2364557819

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/915?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `70.58824%` with `5 lines` in your changes missing coverage. Please review.
   > Project coverage is 34.05%. Comparing base [(`fa275f1`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/fa275f1c007cf3cce60272367d7c952bc867570c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`2757186`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/27571868e466b0cee8a354fc7d7b75650b38aa19?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 1 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/915?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...ala/org/apache/spark/sql/comet/CometScanExec.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/915?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fcomet%2FCometScanExec.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvY29tZXQvQ29tZXRTY2FuRXhlYy5zY2FsYQ==) | 70.58% | [0 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/915?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##               main     #915      +/-   ##
   ============================================
   - Coverage     34.16%   34.05%   -0.12%     
   + Complexity      880      879       -1     
   ============================================
     Files           112      112              
     Lines         43286    43301      +15     
     Branches       9572     9578       +6     
   ============================================
   - Hits          14789    14745      -44     
   - Misses        25478    25518      +40     
   - Partials       3019     3038      +19     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/datafusion-comet/pull/915/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [](https://app.codecov.io/gh/apache/datafusion-comet/pull/915/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.05% <70.58%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/915?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2350699556

   > > Is the change in the `metadata`? If so, should we fix the metadata instead?
   > 
   > Are you referring to the compilation error or the change in explain output?
   
   I meant `CometScanExec. metadata` inherited from `DataSourceScanExec`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "Kimahriman (via GitHub)" <gi...@apache.org>.
Kimahriman commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2332133190

   Oof this breaks a lot of explain plan comparison tests. If this change is ok I can try to work on updating them


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2363851480

   @Kimahriman would you be able to rebase this PR so that we can merge it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "Kimahriman (via GitHub)" <gi...@apache.org>.
Kimahriman commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2364731723

   Done, I think CI failures are unrelated, looks like failures downloading dependencies in Hive tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "Kimahriman (via GitHub)" <gi...@apache.org>.
Kimahriman commented on code in PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#discussion_r1759619909


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometScanExec.scala:
##########
@@ -141,8 +141,30 @@ case class CometScanExec(
     if (wrapped == null) Map.empty else wrapped.metadata
 
   override def verboseStringWithOperatorId(): String = {
-    getTagValue(QueryPlan.OP_ID_TAG).foreach(id => wrapped.setTagValue(QueryPlan.OP_ID_TAG, id))
-    wrapped.verboseStringWithOperatorId()
+    val metadataStr = metadata.toSeq.sorted

Review Comment:
   Just removing setting the `OP_ID_TAG` leads to a formatted explain of:
   
   ```
   == Physical Plan ==
   * ColumnarToRow (2)
   +- CometScan parquet  (1)
   
   
   (unknown) Scan parquet 
   Output [20]: [_1#0, _2#1, _3#2, _4#3, _5#4L, _6#5, _7#6, _8#7, _9#8, _10#9, _11#10L, _12#11, _13#12, _14#13, _15#14, _16#15, _17#16, _18#17, _19#18, _20#19]
   Batched: true
   Location: InMemoryFileIndex [file:/Users/abinford/projects/arrow-datafusion-comet/spark/target/tmp/spark-c0b82b7c-3de1-431b-96bf-56cb37f3a463/test.parquet]
   ReadSchema: struct<_1:boolean,_2:tinyint,_3:smallint,_4:int,_5:bigint,_6:float,_7:double,_8:string,_9:smallint,_10:int,_11:bigint,_12:decimal(20,0),_13:string,_14:binary,_15:decimal(5,2),_16:decimal(18,10),_17:decimal(38,37),_18:timestamp,_19:timestamp,_20:date>
   
   (2) ColumnarToRow [codegen id : 1]
   Input [20]: [_1#0, _2#1, _3#2, _4#3, _5#4L, _6#5, _7#6, _8#7, _9#8, _10#9, _11#10L, _12#11, _13#12, _14#13, _15#14, _16#15, _17#16, _18#17, _19#18, _20#19]
   ```
   
   which is the whole reason it was added in the first place, to fix the unknown operator ID https://github.com/apache/datafusion-comet/commit/bc35fa5a415f4a137177ab4fc66bd0167f9f23d3



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#discussion_r1759611817


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometScanExec.scala:
##########
@@ -141,8 +141,30 @@ case class CometScanExec(
     if (wrapped == null) Map.empty else wrapped.metadata
 
   override def verboseStringWithOperatorId(): String = {
-    getTagValue(QueryPlan.OP_ID_TAG).foreach(id => wrapped.setTagValue(QueryPlan.OP_ID_TAG, id))
-    wrapped.verboseStringWithOperatorId()
+    val metadataStr = metadata.toSeq.sorted

Review Comment:
   Since the original issue was that `OP_ID_TAG` has been removed, is it not sufficient to just remove the offending line 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "Kimahriman (via GitHub)" <gi...@apache.org>.
Kimahriman commented on code in PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#discussion_r1759620402


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometScanExec.scala:
##########
@@ -141,8 +141,30 @@ case class CometScanExec(
     if (wrapped == null) Map.empty else wrapped.metadata
 
   override def verboseStringWithOperatorId(): String = {
-    getTagValue(QueryPlan.OP_ID_TAG).foreach(id => wrapped.setTagValue(QueryPlan.OP_ID_TAG, id))
-    wrapped.verboseStringWithOperatorId()
+    val metadataStr = metadata.toSeq.sorted

Review Comment:
   Plus by what I could see, every other operator is prefixed with `Comet` in the formatted explain, so it's weird for Scan to be the one thing that doesn't actually match up to the physical plan



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "Kimahriman (via GitHub)" <gi...@apache.org>.
Kimahriman commented on PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#issuecomment-2344952848

   > Is the change in the `metadata`? If so, should we fix the metadata instead?
   
   Are you referring to the compilation error or the change in explain output?
   
   > And perhaps we can try to upgrade the Comet dependency to Spark 3.5.2 (separately)
   
   Agreed. I thought about doing that here but I wasn't sure the best way to go about updating the diff for the Spark SQL tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: CometScanExec on Spark 3.5.2 [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #915:
URL: https://github.com/apache/datafusion-comet/pull/915#discussion_r1761594517


##########
spark/src/main/scala/org/apache/spark/sql/comet/CometScanExec.scala:
##########
@@ -141,8 +141,30 @@ case class CometScanExec(
     if (wrapped == null) Map.empty else wrapped.metadata
 
   override def verboseStringWithOperatorId(): String = {
-    getTagValue(QueryPlan.OP_ID_TAG).foreach(id => wrapped.setTagValue(QueryPlan.OP_ID_TAG, id))
-    wrapped.verboseStringWithOperatorId()
+    val metadataStr = metadata.toSeq.sorted

Review Comment:
   I see, that makes sense.
   On an unrelated note, wonder why `CometScanExec` extends `DataSourceScanExec` instead of `FileSourceScanExec` (if it had we would have got the verboseString for free)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org