You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/12 03:01:13 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request, #37161: [SPARK-39748][SQL][SS] Include the origin logical plan for LogicalRDD if it comes from DataFrame
HeartSaVioR opened a new pull request, #37161:
URL: https://github.com/apache/spark/pull/37161
### What changes were proposed in this pull request?
This PR proposes to include the origin logical plan for LogicalRDD, if the LogicalRDD is built from DataFrame's RDD. Once the origin logical plan is available, LogicalRDD produces the stats from origin logical plan rather than default one.
Also, this PR applies the change to ForeachBatchSink, which seems to be the only case as of now in current codebase.
### Why are the changes needed?
The origin logical plan can be useful for several use cases, including:
1. wants to connect the two split logical plans into one (consider the case of foreachBatch sink: origin logical plan represents the plan for streaming query, and the logical plan for new Dataset represents the plan for batch query in user function)
2. inherits plan stats from origin logical plan
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37161: [SPARK-39748][SQL][SS] Include the origin logical plan for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37161:
URL: https://github.com/apache/spark/pull/37161#issuecomment-1181504913
The change is quite simple and straightforward so I just go with merging with 1 approval. That said, I'm open for post-reviewing.
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #37161: [SPARK-39748][SQL][SS] Include the origin logical plan for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #37161:
URL: https://github.com/apache/spark/pull/37161#discussion_r919505305
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -705,6 +705,7 @@ class Dataset[T] private[sql](
LogicalRDD(
logicalPlan.output,
internalRdd,
+ None,
Review Comment:
That is a silly missing point. Thanks for finding this out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #37161: [SPARK-39748][SQL][SS] Include the origin logical plan for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #37161:
URL: https://github.com/apache/spark/pull/37161#discussion_r919518440
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -705,6 +705,7 @@ class Dataset[T] private[sql](
LogicalRDD(
logicalPlan.output,
internalRdd,
+ None,
Review Comment:
#37167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a diff in pull request #37161: [SPARK-39748][SQL][SS] Include the origin logical plan for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #37161:
URL: https://github.com/apache/spark/pull/37161#discussion_r919204026
##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -705,6 +705,7 @@ class Dataset[T] private[sql](
LogicalRDD(
logicalPlan.output,
internalRdd,
+ None,
Review Comment:
why not set the logical plan here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37161: [SPARK-39748][SQL][SS] Include the origin logical plan for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37161:
URL: https://github.com/apache/spark/pull/37161#issuecomment-1181260007
cc. @cloud-fan @viirya Please take a look. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #37161: [SPARK-39748][SQL][SS] Include the origin logical plan for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #37161: [SPARK-39748][SQL][SS] Include the origin logical plan for LogicalRDD if it comes from DataFrame
URL: https://github.com/apache/spark/pull/37161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org