You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "yew1eb (via GitHub)" <gi...@apache.org> on 2025/11/20 11:45:38 UTC
[PR] [AURON #1642] Align NativeFileSourceScanBase.verboseStringWithOperatorId with vanilla Spark [auron]
yew1eb opened a new pull request, #1643:
URL: https://github.com/apache/auron/pull/1643
# Which issue does this PR close?
Closes #1642
# Rationale for this change
# What changes are included in this PR?
override verboseStringWithOperatorId method for NativeFileSourceScanBase to match vanilla spark.
# Are there any user-facing changes?
# How was this patch tested?
Before:
<img width="1469" height="123" alt="Pasted Graphic 2" src="https://github.com/user-attachments/assets/36bed994-9166-4093-ab4e-924dc78a0d2c" />
After:
<img width="1111" height="130" alt="Pasted Graphic 1" src="https://github.com/user-attachments/assets/2259c4ce-aac2-4ce7-b9fe-6b90e9c23b14" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1665] Override verboseStringWithOperatorId in NativeFileSourceScanBase [auron]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on code in PR #1643:
URL: https://github.com/apache/auron/pull/1643#discussion_r2570969926
##########
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeFileSourceScanBase.scala:
##########
@@ -154,4 +152,38 @@ abstract class NativeFileSourceScanBase(basedFileScan: FileSourceScanExec)
}
override protected def doCanonicalize(): SparkPlan = basedFileScan.canonicalized
+
+ /**
+ * Match Spark’s DataSourceScanExec verbose output for consistency.
+ * @see
+ * <a
+ * href="https://github.com/apache/spark/blob/65c3d1cb18c45528d8090ac905d87a8dcd779aa7/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L426-L449">Spark
+ * DataSourceScanExec</a>
+ */
+ override def verboseStringWithOperatorId(): String = {
+ val metadataStr = basedFileScan.metadata.toSeq.sorted
+ .filterNot {
Review Comment:
nit: How about call super.verboseStringWithOperatorId and replace the nodeName, it might be simpler. The code here seems a little complicated which is almost identical to super.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1665] Override verboseStringWithOperatorId in NativeFileSourceScanBase [auron]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1643:
URL: https://github.com/apache/auron/pull/1643#discussion_r2570605022
##########
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeFileSourceScanBase.scala:
##########
@@ -154,4 +152,31 @@ abstract class NativeFileSourceScanBase(basedFileScan: FileSourceScanExec)
}
override protected def doCanonicalize(): SparkPlan = basedFileScan.canonicalized
+
+ override def verboseStringWithOperatorId(): String = {
Review Comment:
Could you add the reference url of spark in comment for better understanding ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1642] Align NativeFileSourceScanBase.verboseStringWithOperatorId with vanilla Spark [auron]
Posted by "yew1eb (via GitHub)" <gi...@apache.org>.
yew1eb commented on PR #1643:
URL: https://github.com/apache/auron/pull/1643#issuecomment-3578699736
@merrily01 @richox the NativeFileSourceScan's explain ouput currently diverges from vanilla spark and forces all metadata into a truncated "Arguments" line that loses information. this change mirrors vanilla spark' formatting exactly, so it have no side effects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1665] Override verboseStringWithOperatorId in NativeFileSourceScanBase [auron]
Posted by "richox (via GitHub)" <gi...@apache.org>.
richox merged PR #1643:
URL: https://github.com/apache/auron/pull/1643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #1665] Override verboseStringWithOperatorId in NativeFileSourceScanBase [auron]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1643:
URL: https://github.com/apache/auron/pull/1643#discussion_r2570988984
##########
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeFileSourceScanBase.scala:
##########
@@ -154,4 +152,38 @@ abstract class NativeFileSourceScanBase(basedFileScan: FileSourceScanExec)
}
override protected def doCanonicalize(): SparkPlan = basedFileScan.canonicalized
+
+ /**
+ * Match Spark’s DataSourceScanExec verbose output for consistency.
+ * @see
+ * <a
+ * href="https://github.com/apache/spark/blob/65c3d1cb18c45528d8090ac905d87a8dcd779aa7/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L426-L449">Spark
+ * DataSourceScanExec</a>
+ */
+ override def verboseStringWithOperatorId(): String = {
+ val metadataStr = basedFileScan.metadata.toSeq.sorted
+ .filterNot {
Review Comment:
Could you directly use the upstream method? If so, it's pretty nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org