You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nan Zhu <zh...@gmail.com> on 2014/02/27 17:13:14 UTC
test cases stuck on "local-cluster mode" of ReplSuite?
Hi, all
Actually this problem exists for months in my side, when I run the test cases, it will stop (actually pause?) at the ReplSuite
[info] ReplSuite:
2014-02-27 10:57:37.220 java[3911:1303] Unable to load realm info from SCDynamicStore
[info] - propagation of local properties (7 seconds, 646 milliseconds)
[info] - simple foreach with accumulator (6 seconds, 204 milliseconds)
[info] - external vars (4 seconds, 271 milliseconds)
[info] - external classes (3 seconds, 186 milliseconds)
[info] - external functions (4 seconds, 843 milliseconds)
[info] - external functions that access vars (3 seconds, 503 milliseconds)
[info] - broadcast vars (4 seconds, 313 milliseconds)
[info] - interacting with files (2 seconds, 492 milliseconds)
The next test case should be
test("local-cluster mode") {
val output = runInterpreter("local-cluster[1,1,512]",
"""
|var v = 7
|def getV() = v
|sc.parallelize(1 to 10).map(x => getV()).collect.reduceLeft(_+_)
|v = 10
|sc.parallelize(1 to 10).map(x => getV()).collect.reduceLeft(_+_)
|var array = new Array[Int](5)
|val broadcastArray = sc.broadcast(array)
|sc.parallelize(0 to 4).map(x => broadcastArray.value(x)).collect
|array(0) = 5
|sc.parallelize(0 to 4).map(x => broadcastArray.value(x)).collect
""".stripMargin)
assertDoesNotContain("error:", output)
assertDoesNotContain("Exception", output)
assertContains("res0: Int = 70", output)
assertContains("res1: Int = 100", output)
assertContains("res2: Array[Int] = Array(0, 0, 0, 0, 0)", output)
assertContains("res4: Array[Int] = Array(0, 0, 0, 0, 0)", output)
}
I didn’t see any reason for it spending so much time on it….
Any idea? I’m using mbp, OS X 10.9.1, Intel Core i7 2.9 GHz, Memory 8GB 1600 MHz DDR3
Best,
--
Nan Zhu
Re: test cases stuck on "local-cluster mode" of ReplSuite?
Posted by Nan Zhu <zh...@gmail.com>.
Yeah,
I tested that, I had my SPARK_HOME point to a very old location, after I fixed that, everything goes well
Thank you so much for pointing this out
Best,
--
Nan Zhu
On Friday, March 14, 2014 at 6:41 PM, Michael Armbrust wrote:
> Sorry to revive an old thread, but I just ran into this issue myself. It
> is likely that you do not have the assembly jar built, or that you have
> SPARK_HOME set incorrectly (it does not need to be set).
>
> Michael
>
>
> On Thu, Feb 27, 2014 at 8:13 AM, Nan Zhu <zhunanmcgill@gmail.com (mailto:zhunanmcgill@gmail.com)> wrote:
>
> > Hi, all
> >
> > Actually this problem exists for months in my side, when I run the test
> > cases, it will stop (actually pause?) at the ReplSuite
> >
> > [info] ReplSuite:
> > 2014-02-27 10:57:37.220 java[3911:1303] Unable to load realm info from
> > SCDynamicStore
> > [info] - propagation of local properties (7 seconds, 646 milliseconds)
> > [info] - simple foreach with accumulator (6 seconds, 204 milliseconds)
> > [info] - external vars (4 seconds, 271 milliseconds)
> > [info] - external classes (3 seconds, 186 milliseconds)
> > [info] - external functions (4 seconds, 843 milliseconds)
> > [info] - external functions that access vars (3 seconds, 503 milliseconds)
> > [info] - broadcast vars (4 seconds, 313 milliseconds)
> > [info] - interacting with files (2 seconds, 492 milliseconds)
> >
> >
> >
> > The next test case should be
> >
> > test("local-cluster mode") {
> > val output = runInterpreter("local-cluster[1,1,512]",
> > """
> > |var v = 7
> > |def getV() = v
> > |sc.parallelize(1 to 10).map(x => getV()).collect.reduceLeft(_+_)
> > |v = 10
> > |sc.parallelize(1 to 10).map(x => getV()).collect.reduceLeft(_+_)
> > |var array = new Array[Int](5)
> > |val broadcastArray = sc.broadcast(array)
> > |sc.parallelize(0 to 4).map(x => broadcastArray.value(x)).collect
> > |array(0) = 5
> > |sc.parallelize(0 to 4).map(x => broadcastArray.value(x)).collect
> > """.stripMargin)
> > assertDoesNotContain("error:", output)
> > assertDoesNotContain("Exception", output)
> > assertContains("res0: Int = 70", output)
> > assertContains("res1: Int = 100", output)
> > assertContains("res2: Array[Int] = Array(0, 0, 0, 0, 0)", output)
> > assertContains("res4: Array[Int] = Array(0, 0, 0, 0, 0)", output)
> > }
> >
> >
> >
> > I didn't see any reason for it spending so much time on it....
> >
> > Any idea? I'm using mbp, OS X 10.9.1, Intel Core i7 2.9 GHz, Memory 8GB
> > 1600 MHz DDR3
> >
> > Best,
> >
> > --
> > Nan Zhu
> >
>
>
>
Re: test cases stuck on "local-cluster mode" of ReplSuite?
Posted by Michael Armbrust <mi...@databricks.com>.
Sorry to revive an old thread, but I just ran into this issue myself. It
is likely that you do not have the assembly jar built, or that you have
SPARK_HOME set incorrectly (it does not need to be set).
Michael
On Thu, Feb 27, 2014 at 8:13 AM, Nan Zhu <zh...@gmail.com> wrote:
> Hi, all
>
> Actually this problem exists for months in my side, when I run the test
> cases, it will stop (actually pause?) at the ReplSuite
>
> [info] ReplSuite:
> 2014-02-27 10:57:37.220 java[3911:1303] Unable to load realm info from
> SCDynamicStore
> [info] - propagation of local properties (7 seconds, 646 milliseconds)
> [info] - simple foreach with accumulator (6 seconds, 204 milliseconds)
> [info] - external vars (4 seconds, 271 milliseconds)
> [info] - external classes (3 seconds, 186 milliseconds)
> [info] - external functions (4 seconds, 843 milliseconds)
> [info] - external functions that access vars (3 seconds, 503 milliseconds)
> [info] - broadcast vars (4 seconds, 313 milliseconds)
> [info] - interacting with files (2 seconds, 492 milliseconds)
>
>
>
> The next test case should be
>
> test("local-cluster mode") {
> val output = runInterpreter("local-cluster[1,1,512]",
> """
> |var v = 7
> |def getV() = v
> |sc.parallelize(1 to 10).map(x => getV()).collect.reduceLeft(_+_)
> |v = 10
> |sc.parallelize(1 to 10).map(x => getV()).collect.reduceLeft(_+_)
> |var array = new Array[Int](5)
> |val broadcastArray = sc.broadcast(array)
> |sc.parallelize(0 to 4).map(x => broadcastArray.value(x)).collect
> |array(0) = 5
> |sc.parallelize(0 to 4).map(x => broadcastArray.value(x)).collect
> """.stripMargin)
> assertDoesNotContain("error:", output)
> assertDoesNotContain("Exception", output)
> assertContains("res0: Int = 70", output)
> assertContains("res1: Int = 100", output)
> assertContains("res2: Array[Int] = Array(0, 0, 0, 0, 0)", output)
> assertContains("res4: Array[Int] = Array(0, 0, 0, 0, 0)", output)
> }
>
>
>
> I didn't see any reason for it spending so much time on it....
>
> Any idea? I'm using mbp, OS X 10.9.1, Intel Core i7 2.9 GHz, Memory 8GB
> 1600 MHz DDR3
>
> Best,
>
> --
> Nan Zhu
>
>