You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 诺铁 <no...@gmail.com> on 2014/09/16 09:06:20 UTC
SparkContext creation slow down unit tests
hi,
I am trying to write some unit test, following spark programming guide
<http://spark.apache.org/docs/latest/programming-guide.html#unit-testing>.
but I observed unit test runs very slow(the code is just a SparkPi), so I
turn log level to trace and look through the log output. and found
creation of SparkContext seems to take most time.
there are some action take a lot of time:
1, seems starting jetty
14:04:55.038 [ScalaTest-run-running-MySparkPiSpec] DEBUG
o.e.jetty.servlet.ServletHandler -
servletNameMap={org.apache.spark.ui.JettyUtils$$anon$1-672f11c2=org.apache.spark.ui.JettyUtils$$anon$1-672f11c2}
14:05:25.121 [ScalaTest-run-running-MySparkPiSpec] DEBUG
o.e.jetty.util.component.Container - Container
org.eclipse.jetty.server.Server@e3cee7b +
SelectChannelConnector@0.0.0.0:4040 as connector
2, I don't know what's this
14:05:25.202 [ScalaTest-run-running-MySparkPiSpec] DEBUG
org.apache.hadoop.security.Groups - Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
14:05:54.594 [spark-akka.actor.default-dispatcher-4] TRACE
o.a.s.s.BlockManagerMasterActor - Checking for hosts with no recent heart
beats in BlockManagerMaster.
are there any way to make this faster for unit test?
I also notice in spark's unit test, that there exists a SharedSparkContext
<https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/SharedSparkContext.scala>,
is this the suggested way to work around this problem? if so, I would
suggest to document this in programming guide.
Re: SparkContext creation slow down unit tests
Posted by 诺铁 <no...@gmail.com>.
sorry for disturb, please ignore this mail....
in the end, I find it slow because lack of memory in my machine..
sorry again.
On Tue, Sep 16, 2014 at 3:26 PM, 诺铁 <no...@gmail.com> wrote:
> I connect my sample project to a hosted CI service, it only takes 3
> seconds to run there...while the same tests takes 2minutes on my macbook
> pro. so maybe this is a mac os specific problem?
>
> On Tue, Sep 16, 2014 at 3:06 PM, 诺铁 <no...@gmail.com> wrote:
>
>> hi,
>>
>> I am trying to write some unit test, following spark programming guide
>> <http://spark.apache.org/docs/latest/programming-guide.html#unit-testing>
>> .
>> but I observed unit test runs very slow(the code is just a SparkPi), so
>> I turn log level to trace and look through the log output. and found
>> creation of SparkContext seems to take most time.
>>
>> there are some action take a lot of time:
>> 1, seems starting jetty
>> 14:04:55.038 [ScalaTest-run-running-MySparkPiSpec] DEBUG
>> o.e.jetty.servlet.ServletHandler -
>> servletNameMap={org.apache.spark.ui.JettyUtils$$anon$1-672f11c2=org.apache.spark.ui.JettyUtils$$anon$1-672f11c2}
>> 14:05:25.121 [ScalaTest-run-running-MySparkPiSpec] DEBUG
>> o.e.jetty.util.component.Container - Container
>> org.eclipse.jetty.server.Server@e3cee7b +
>> SelectChannelConnector@0.0.0.0:4040 as connector
>>
>> 2, I don't know what's this
>> 14:05:25.202 [ScalaTest-run-running-MySparkPiSpec] DEBUG
>> org.apache.hadoop.security.Groups - Group mapping
>> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
>> cacheTimeout=300000
>> 14:05:54.594 [spark-akka.actor.default-dispatcher-4] TRACE
>> o.a.s.s.BlockManagerMasterActor - Checking for hosts with no recent heart
>> beats in BlockManagerMaster.
>>
>>
>> are there any way to make this faster for unit test?
>> I also notice in spark's unit test, that there exists a
>> SharedSparkContext
>> <https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/SharedSparkContext.scala>,
>> is this the suggested way to work around this problem? if so, I would
>> suggest to document this in programming guide.
>>
>
>
Re: SparkContext creation slow down unit tests
Posted by 诺铁 <no...@gmail.com>.
I connect my sample project to a hosted CI service, it only takes 3 seconds
to run there...while the same tests takes 2minutes on my macbook pro. so
maybe this is a mac os specific problem?
On Tue, Sep 16, 2014 at 3:06 PM, 诺铁 <no...@gmail.com> wrote:
> hi,
>
> I am trying to write some unit test, following spark programming guide
> <http://spark.apache.org/docs/latest/programming-guide.html#unit-testing>
> .
> but I observed unit test runs very slow(the code is just a SparkPi), so I
> turn log level to trace and look through the log output. and found
> creation of SparkContext seems to take most time.
>
> there are some action take a lot of time:
> 1, seems starting jetty
> 14:04:55.038 [ScalaTest-run-running-MySparkPiSpec] DEBUG
> o.e.jetty.servlet.ServletHandler -
> servletNameMap={org.apache.spark.ui.JettyUtils$$anon$1-672f11c2=org.apache.spark.ui.JettyUtils$$anon$1-672f11c2}
> 14:05:25.121 [ScalaTest-run-running-MySparkPiSpec] DEBUG
> o.e.jetty.util.component.Container - Container
> org.eclipse.jetty.server.Server@e3cee7b +
> SelectChannelConnector@0.0.0.0:4040 as connector
>
> 2, I don't know what's this
> 14:05:25.202 [ScalaTest-run-running-MySparkPiSpec] DEBUG
> org.apache.hadoop.security.Groups - Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=300000
> 14:05:54.594 [spark-akka.actor.default-dispatcher-4] TRACE
> o.a.s.s.BlockManagerMasterActor - Checking for hosts with no recent heart
> beats in BlockManagerMaster.
>
>
> are there any way to make this faster for unit test?
> I also notice in spark's unit test, that there exists a SharedSparkContext
> <https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/SharedSparkContext.scala>,
> is this the suggested way to work around this problem? if so, I would
> suggest to document this in programming guide.
>