[GitHub] [spark] sririshindra commented on a change in pull request #31477: [SPARK-34369][SQL][WEBUI] Track number of pairs processed out of Join.

[GitBox <gi...@apache.org>]

sririshindra commented on a change in pull request #31477:
URL: https://github.com/apache/spark/pull/31477#discussion_r570672502



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala
##########
@@ -89,13 +90,20 @@ case class BroadcastNestedLoopJoinExec(
     }
   }
 
-  @transient private lazy val boundCondition = {
+  private val numMatchedPairs = longMetric("numMatchedPairs")
+
+  @transient private lazy val boundCondition: InternalRow => Boolean =
     if (condition.isDefined) {

Review comment:
       If the condition is empty, then `numOutputRows == numMatchedPairs` only in the case of InnerJoin. It is not necessarily the case for other types of join, because numMatchedPairs is only measuring the records when there is a common key on both the build side and the stream side.  For instance, in the case of leftOuterJoin `numOutputRows  = numMatchedPairs  + (number of non matched pairs on the left side of the join)`.
   Moreover, if we define a metric and fail to populate it for some cases, then the existing unit tests in the SQLMetricsSuite will fail because of the way the unit tests fetch the actual metrics. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org