You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/16 03:57:43 UTC
[GitHub] [spark] WangGuangxin commented on a change in pull request #26899:
[SPARK-28332][SQL] Reserve init value -1 only when do min max statistics in
SQLMetrics
WangGuangxin commented on a change in pull request #26899: [SPARK-28332][SQL] Reserve init value -1 only when do min max statistics in SQLMetrics
URL: https://github.com/apache/spark/pull/26899#discussion_r358044877
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
##########
@@ -173,7 +173,12 @@ class SQLAppStatusListener(
event.taskMetrics.externalAccums.flatMap { a =>
// This call may fail if the accumulator is gc'ed, so account for that.
try {
- Some(a.toInfo(Some(a.value), None))
+ val accumValue = if (a.isInstanceOf[SQLMetric]) {
Review comment:
As explained in https://issues.apache.org/jira/browse/SPARK-11013
> when we call RDD API inside SparkPlan, we are very likely to reference the SparkPlan in the closure and thus serialize and transfer a SparkPlan tree to executor side. When we deserialize it, the accumulators in child SparkPlan are also deserialized and registered, and always report zero value.
> This is not a problem currently because we only have one operation to aggregate the accumulators: add. However, if we wanna support more complex metric like min, the extra zero values will lead to wrong result.
Use -1 as init value here is want to distinguish whether a SQLMetric is initialized or not, so that we can filter out those irrelevant/uninitialized metrics when do min/max statistics.
If we use 0 as init value, we cannot tell whether it is uninitialized or it is zero as it is
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org