You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/09/15 08:38:17 UTC
spark git commit: [SPARK-17521] Error when I use
sparkContext.makeRDD(Seq())
Repository: spark
Updated Branches:
refs/heads/master f893e2625 -> 647ee05e5
[SPARK-17521] Error when I use sparkContext.makeRDD(Seq())
## What changes were proposed in this pull request?
when i use sc.makeRDD below
```
val data3 = sc.makeRDD(Seq())
println(data3.partitions.length)
```
I got an error:
Exception in thread "main" java.lang.IllegalArgumentException: Positive number of slices required
We can fix this bug just modify the last line ,do a check of seq.size
```
def makeRDD[T: ClassTag](seq: Seq[(T, Seq[String])]): RDD[T] = withScope {
assertNotStopped()
val indexToPrefs = seq.zipWithIndex.map(t => (t._2, t._1._2)).toMap
new ParallelCollectionRDD[T](this, seq.map(_._1), math.max(seq.size, defaultParallelism), indexToPrefs)
}
```
## How was this patch tested?
manual tests
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Author: codlife <10...@qq.com>
Author: codlife <wa...@otcaix.iscas.ac.cn>
Closes #15077 from codlife/master.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/647ee05e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/647ee05e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/647ee05e
Branch: refs/heads/master
Commit: 647ee05e5815bde361662a9286ac602c44b4d4e6
Parents: f893e26
Author: codlife <10...@qq.com>
Authored: Thu Sep 15 09:38:13 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Thu Sep 15 09:38:13 2016 +0100
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/SparkContext.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/647ee05e/core/src/main/scala/org/apache/spark/SparkContext.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index e32e4aa..35b6334 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -795,7 +795,7 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
def makeRDD[T: ClassTag](seq: Seq[(T, Seq[String])]): RDD[T] = withScope {
assertNotStopped()
val indexToPrefs = seq.zipWithIndex.map(t => (t._2, t._1._2)).toMap
- new ParallelCollectionRDD[T](this, seq.map(_._1), seq.size, indexToPrefs)
+ new ParallelCollectionRDD[T](this, seq.map(_._1), math.max(seq.size, 1), indexToPrefs)
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org