You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2017/03/15 22:01:19 UTC
spark git commit: [MINOR][CORE] Fix a info message of
`prunePartitions`
Repository: spark
Updated Branches:
refs/heads/master 97cc5e5a5 -> 54a3697f1
[MINOR][CORE] Fix a info message of `prunePartitions`
## What changes were proposed in this pull request?
`PrunedInMemoryFileIndex.prunePartitions` shows `pruned NaN% partitions` for the following case.
```scala
scala> Seq.empty[(String, String)].toDF("a", "p").write.partitionBy("p").saveAsTable("t1")
scala> sc.setLogLevel("INFO")
scala> spark.table("t1").filter($"p" === "1").select($"a").show
...
17/03/13 00:33:04 INFO PrunedInMemoryFileIndex: Selected 0 partitions out of 0, pruned NaN% partitions.
```
After this PR, the message looks like this.
```scala
17/03/15 10:39:48 INFO PrunedInMemoryFileIndex: Selected 0 partitions out of 0, pruned 0 partitions.
```
## How was this patch tested?
Pass the Jenkins with the existing tests.
Author: Dongjoon Hyun <do...@apache.org>
Closes #17273 from dongjoon-hyun/SPARK-EMPTY-PARTITION.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/54a3697f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/54a3697f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/54a3697f
Branch: refs/heads/master
Commit: 54a3697f1fb562ef9ed8fed9caffc62b84763049
Parents: 97cc5e5
Author: Dongjoon Hyun <do...@apache.org>
Authored: Wed Mar 15 15:01:16 2017 -0700
Committer: Reynold Xin <rx...@databricks.com>
Committed: Wed Mar 15 15:01:16 2017 -0700
----------------------------------------------------------------------
.../sql/execution/datasources/PartitioningAwareFileIndex.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/54a3697f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
index a5fa8b3..db8bbc5 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
@@ -186,7 +186,8 @@ abstract class PartitioningAwareFileIndex(
val total = partitions.length
val selectedSize = selected.length
val percentPruned = (1 - selectedSize.toDouble / total.toDouble) * 100
- s"Selected $selectedSize partitions out of $total, pruned $percentPruned% partitions."
+ s"Selected $selectedSize partitions out of $total, " +
+ s"pruned ${if (total == 0) "0" else s"$percentPruned%"} partitions."
}
selected
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org