You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Matthew Cheah <ma...@gmail.com> on 2014/10/15 03:17:14 UTC
Unit testing Master-Worker Message Passing
Hi everyone,
I’m adding some new message passing between the Master and Worker actors in
order to address https://issues.apache.org/jira/browse/SPARK-3736 .
I was wondering if these kinds of interactions are tested in the automated
Jenkins test suite, and if so, where I could find some examples to help me
do the same.
Thanks!
-Matt Cheah
Re: Unit testing Master-Worker Message Passing
Posted by Josh Rosen <ro...@gmail.com>.
Hi Matt,
I’m not sure whether those tests will actually find this specific issue. The tests that I linked to test Spark’s Zookeeper-based multi-master mode, whereas it sounds like you’re seeing this issue in regular standalone cluster. In those tests, the workers disconnect from the master because we've actually killed the master, whereas it sounds like your workers are disconnecting due to intermittent network issues and you’d like them to reconnect to the same master.
That FaultToleranceSuite is not run automatically and I don’t think that it was designed for automatic use: for example, it doesn’t clean up Docker containers after failed tests, doesn’t use a framework like ScalaTest for structured test result reporting, etc. The new framework that I’m working on will address these issues.
- Josh
On October 15, 2014 at 3:04:42 PM, Matthew Cheah (matthew.c.cheah@gmail.com) wrote:
Thanks Josh! These tests seem to cover the cases I'm looking for already =).
What's interesting though is that we still ran into SPARK-3736 despite such integration tests being in place to catch it - specifically, the case when the master disconnects and reconnects, the workers should reconnect to the master after the master restarts. Are the tests here run regularly, i.e. Jenkins build or nightly build, and if so how did that test case pass while SPARK-3736 apparently still exists?
At any rate, I think I'll submit my fix PR but with no particular extra automated test written for it, since it seems like FaultToleranceTest sufficiently covers what I need.
On Wed, Oct 15, 2014 at 2:42 PM, Josh Rosen <ro...@gmail.com> wrote:
There are some end-to-end integration tests of Master <-> Worker fault-tolerance in https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala
I’ve actually been working to develop a more generalized Docker-based integration-testing framework for Spark in order to test Master <-> Worker interactions. I’d like to eventually clean up my code and release it publicly.
On October 15, 2014 at 2:39:22 PM, Matthew Cheah (matthew.c.cheah@gmail.com) wrote:
I think on a higher level I also want to ask why such unit testing has not
actually been done in this codebase. If it's not a common practice to test
message passing then I'm fine with leaving out the unit test, however I'm
more curious as to why such testing was not done before.
On Wed, Oct 15, 2014 at 2:18 PM, Chester Chen <ch...@alpinenow.com> wrote:
> You can call resolve method on ActorSelection.resolveOne() to see if the
> actor is still there or the path is correct. The method returns a future
> and you can wait for it with timeout. This way, you know the actor is live
> or already dead or incorrect.
>
> Another way, is to send Identify method to ActorSystem, if it returns with
> correct identified message; then you can act on it, otherwise, ...
>
> hope this helps
>
> Chester
>
> On Wed, Oct 15, 2014 at 1:38 PM, Matthew Cheah <ma...@gmail.com>
> wrote:
>
>> What's happening when I do this is that the Worker tries to get the Master
>> actor by calling context.actorSelection(), and the RegisterWorker message
>> gets sent to the dead letters mailbox instead of being picked up by
>> expectMsg. I'm new to Akka and I've tried various ways to registering a
>> "mock" master to no avail.
>>
>> I would think there would be at least some kind of test for master -
>> worker
>> message passing, no?
>>
>> On Wed, Oct 15, 2014 at 11:28 AM, Nan Zhu <zh...@gmail.com> wrote:
>>
>> > I don’t think there are test cases for Worker itself
>> >
>> >
>> > You can
>> >
>> >
>> > val actorRef = TestActorRef[Master](Props(classOf[Master], ...))(
>> > actorSystem) actorRef.underlyingActor.receive(Heartbeat)
>>
>> >
>> > and use expectMsg to test if Master can reply correct message by
>> assuming
>> > Worker is absolutely correct
>> >
>> > Then in another test case to test if Worker can send register message to
>> > Master after receiving Master’s “re-register” instruction, (in this test
>> > case assuming that the Master is absolutely right)
>> >
>> > Best,
>> >
>> > --
>> > Nan Zhu
>> >
>> > On Wednesday, October 15, 2014 at 2:04 PM, Matthew Cheah wrote:
>> >
>> > Thanks, the example was helpful.
>> >
>> > However, testing the Worker itself is a lot more complicated than
>> > WorkerWatcher, since the Worker class is quite a bit more complex. Are
>> > there any tests that inspect the Worker itself?
>> >
>> > Thanks,
>> >
>> > -Matt Cheah
>> >
>> > On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu <zh...@gmail.com>
>> wrote:
>> >
>> > You can use akka testkit
>> >
>> > Example:
>> >
>> >
>> >
>> https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
>> >
>> > --
>> > Nan Zhu
>> >
>> > On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
>> >
>> > Hi everyone,
>> >
>> > I’m adding some new message passing between the Master and Worker
>> actors in
>> > order to address https://issues.apache.org/jira/browse/SPARK-3736 .
>> >
>> > I was wondering if these kinds of interactions are tested in the
>> automated
>> > Jenkins test suite, and if so, where I could find some examples to help
>> me
>> > do the same.
>> >
>> > Thanks!
>> >
>> > -Matt Cheah
>> >
>> >
>> >
>> >
>> >
>>
>
>
Re: Unit testing Master-Worker Message Passing
Posted by Matthew Cheah <ma...@gmail.com>.
Thanks Josh! These tests seem to cover the cases I'm looking for already =).
What's interesting though is that we still ran into SPARK-3736 despite such
integration tests being in place to catch it - specifically, the case when
the master disconnects and reconnects, the workers should reconnect to the
master after the master restarts. Are the tests here run regularly, i.e.
Jenkins build or nightly build, and if so how did that test case pass while
SPARK-3736 apparently still exists?
At any rate, I think I'll submit my fix PR but with no particular extra
automated test written for it, since it seems like FaultToleranceTest
sufficiently covers what I need.
On Wed, Oct 15, 2014 at 2:42 PM, Josh Rosen <ro...@gmail.com> wrote:
> There are some end-to-end integration tests of Master <-> Worker
> fault-tolerance in
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala
>
> I’ve actually been working to develop a more generalized Docker-based
> integration-testing framework for Spark in order to test Master <-> Worker
> interactions. I’d like to eventually clean up my code and release it
> publicly.
>
> On October 15, 2014 at 2:39:22 PM, Matthew Cheah (
> matthew.c.cheah@gmail.com) wrote:
>
> I think on a higher level I also want to ask why such unit testing has not
> actually been done in this codebase. If it's not a common practice to test
> message passing then I'm fine with leaving out the unit test, however I'm
> more curious as to why such testing was not done before.
>
> On Wed, Oct 15, 2014 at 2:18 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
>
> > You can call resolve method on ActorSelection.resolveOne() to see if the
> > actor is still there or the path is correct. The method returns a future
> > and you can wait for it with timeout. This way, you know the actor is
> live
> > or already dead or incorrect.
> >
> > Another way, is to send Identify method to ActorSystem, if it returns
> with
> > correct identified message; then you can act on it, otherwise, ...
> >
> > hope this helps
> >
> > Chester
> >
> > On Wed, Oct 15, 2014 at 1:38 PM, Matthew Cheah <
> matthew.c.cheah@gmail.com>
> > wrote:
> >
> >> What's happening when I do this is that the Worker tries to get the
> Master
> >> actor by calling context.actorSelection(), and the RegisterWorker
> message
> >> gets sent to the dead letters mailbox instead of being picked up by
> >> expectMsg. I'm new to Akka and I've tried various ways to registering a
> >> "mock" master to no avail.
> >>
> >> I would think there would be at least some kind of test for master -
> >> worker
> >> message passing, no?
> >>
> >> On Wed, Oct 15, 2014 at 11:28 AM, Nan Zhu <zh...@gmail.com>
> wrote:
> >>
> >> > I don’t think there are test cases for Worker itself
> >> >
> >> >
> >> > You can
> >> >
> >> >
> >> > val actorRef = TestActorRef[Master](Props(classOf[Master], ...))(
> >> > actorSystem) actorRef.underlyingActor.receive(Heartbeat)
> >>
> >> >
> >> > and use expectMsg to test if Master can reply correct message by
> >> assuming
> >> > Worker is absolutely correct
> >> >
> >> > Then in another test case to test if Worker can send register message
> to
> >> > Master after receiving Master’s “re-register” instruction, (in this
> test
> >> > case assuming that the Master is absolutely right)
> >> >
> >> > Best,
> >> >
> >> > --
> >> > Nan Zhu
> >> >
> >> > On Wednesday, October 15, 2014 at 2:04 PM, Matthew Cheah wrote:
> >> >
> >> > Thanks, the example was helpful.
> >> >
> >> > However, testing the Worker itself is a lot more complicated than
> >> > WorkerWatcher, since the Worker class is quite a bit more complex.
> Are
> >> > there any tests that inspect the Worker itself?
> >> >
> >> > Thanks,
> >> >
> >> > -Matt Cheah
> >> >
> >> > On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu <zh...@gmail.com>
> >> wrote:
> >> >
> >> > You can use akka testkit
> >> >
> >> > Example:
> >> >
> >> >
> >> >
> >>
> https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
> >> >
> >> > --
> >> > Nan Zhu
> >> >
> >> > On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
> >> >
> >> > Hi everyone,
> >> >
> >> > I’m adding some new message passing between the Master and Worker
> >> actors in
> >> > order to address https://issues.apache.org/jira/browse/SPARK-3736 .
> >> >
> >> > I was wondering if these kinds of interactions are tested in the
> >> automated
> >> > Jenkins test suite, and if so, where I could find some examples to
> help
> >> me
> >> > do the same.
> >> >
> >> > Thanks!
> >> >
> >> > -Matt Cheah
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
>
>
Re: Unit testing Master-Worker Message Passing
Posted by Josh Rosen <ro...@gmail.com>.
There are some end-to-end integration tests of Master <-> Worker fault-tolerance in https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala
I’ve actually been working to develop a more generalized Docker-based integration-testing framework for Spark in order to test Master <-> Worker interactions. I’d like to eventually clean up my code and release it publicly.
On October 15, 2014 at 2:39:22 PM, Matthew Cheah (matthew.c.cheah@gmail.com) wrote:
I think on a higher level I also want to ask why such unit testing has not
actually been done in this codebase. If it's not a common practice to test
message passing then I'm fine with leaving out the unit test, however I'm
more curious as to why such testing was not done before.
On Wed, Oct 15, 2014 at 2:18 PM, Chester Chen <ch...@alpinenow.com> wrote:
> You can call resolve method on ActorSelection.resolveOne() to see if the
> actor is still there or the path is correct. The method returns a future
> and you can wait for it with timeout. This way, you know the actor is live
> or already dead or incorrect.
>
> Another way, is to send Identify method to ActorSystem, if it returns with
> correct identified message; then you can act on it, otherwise, ...
>
> hope this helps
>
> Chester
>
> On Wed, Oct 15, 2014 at 1:38 PM, Matthew Cheah <ma...@gmail.com>
> wrote:
>
>> What's happening when I do this is that the Worker tries to get the Master
>> actor by calling context.actorSelection(), and the RegisterWorker message
>> gets sent to the dead letters mailbox instead of being picked up by
>> expectMsg. I'm new to Akka and I've tried various ways to registering a
>> "mock" master to no avail.
>>
>> I would think there would be at least some kind of test for master -
>> worker
>> message passing, no?
>>
>> On Wed, Oct 15, 2014 at 11:28 AM, Nan Zhu <zh...@gmail.com> wrote:
>>
>> > I don’t think there are test cases for Worker itself
>> >
>> >
>> > You can
>> >
>> >
>> > val actorRef = TestActorRef[Master](Props(classOf[Master], ...))(
>> > actorSystem) actorRef.underlyingActor.receive(Heartbeat)
>>
>> >
>> > and use expectMsg to test if Master can reply correct message by
>> assuming
>> > Worker is absolutely correct
>> >
>> > Then in another test case to test if Worker can send register message to
>> > Master after receiving Master’s “re-register” instruction, (in this test
>> > case assuming that the Master is absolutely right)
>> >
>> > Best,
>> >
>> > --
>> > Nan Zhu
>> >
>> > On Wednesday, October 15, 2014 at 2:04 PM, Matthew Cheah wrote:
>> >
>> > Thanks, the example was helpful.
>> >
>> > However, testing the Worker itself is a lot more complicated than
>> > WorkerWatcher, since the Worker class is quite a bit more complex. Are
>> > there any tests that inspect the Worker itself?
>> >
>> > Thanks,
>> >
>> > -Matt Cheah
>> >
>> > On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu <zh...@gmail.com>
>> wrote:
>> >
>> > You can use akka testkit
>> >
>> > Example:
>> >
>> >
>> >
>> https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
>> >
>> > --
>> > Nan Zhu
>> >
>> > On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
>> >
>> > Hi everyone,
>> >
>> > I’m adding some new message passing between the Master and Worker
>> actors in
>> > order to address https://issues.apache.org/jira/browse/SPARK-3736 .
>> >
>> > I was wondering if these kinds of interactions are tested in the
>> automated
>> > Jenkins test suite, and if so, where I could find some examples to help
>> me
>> > do the same.
>> >
>> > Thanks!
>> >
>> > -Matt Cheah
>> >
>> >
>> >
>> >
>> >
>>
>
>
Re: Unit testing Master-Worker Message Passing
Posted by Matthew Cheah <ma...@gmail.com>.
I think on a higher level I also want to ask why such unit testing has not
actually been done in this codebase. If it's not a common practice to test
message passing then I'm fine with leaving out the unit test, however I'm
more curious as to why such testing was not done before.
On Wed, Oct 15, 2014 at 2:18 PM, Chester Chen <ch...@alpinenow.com> wrote:
> You can call resolve method on ActorSelection.resolveOne() to see if the
> actor is still there or the path is correct. The method returns a future
> and you can wait for it with timeout. This way, you know the actor is live
> or already dead or incorrect.
>
> Another way, is to send Identify method to ActorSystem, if it returns with
> correct identified message; then you can act on it, otherwise, ...
>
> hope this helps
>
> Chester
>
> On Wed, Oct 15, 2014 at 1:38 PM, Matthew Cheah <ma...@gmail.com>
> wrote:
>
>> What's happening when I do this is that the Worker tries to get the Master
>> actor by calling context.actorSelection(), and the RegisterWorker message
>> gets sent to the dead letters mailbox instead of being picked up by
>> expectMsg. I'm new to Akka and I've tried various ways to registering a
>> "mock" master to no avail.
>>
>> I would think there would be at least some kind of test for master -
>> worker
>> message passing, no?
>>
>> On Wed, Oct 15, 2014 at 11:28 AM, Nan Zhu <zh...@gmail.com> wrote:
>>
>> > I don’t think there are test cases for Worker itself
>> >
>> >
>> > You can
>> >
>> >
>> > val actorRef = TestActorRef[Master](Props(classOf[Master], ...))(
>> > actorSystem) actorRef.underlyingActor.receive(Heartbeat)
>>
>> >
>> > and use expectMsg to test if Master can reply correct message by
>> assuming
>> > Worker is absolutely correct
>> >
>> > Then in another test case to test if Worker can send register message to
>> > Master after receiving Master’s “re-register” instruction, (in this test
>> > case assuming that the Master is absolutely right)
>> >
>> > Best,
>> >
>> > --
>> > Nan Zhu
>> >
>> > On Wednesday, October 15, 2014 at 2:04 PM, Matthew Cheah wrote:
>> >
>> > Thanks, the example was helpful.
>> >
>> > However, testing the Worker itself is a lot more complicated than
>> > WorkerWatcher, since the Worker class is quite a bit more complex. Are
>> > there any tests that inspect the Worker itself?
>> >
>> > Thanks,
>> >
>> > -Matt Cheah
>> >
>> > On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu <zh...@gmail.com>
>> wrote:
>> >
>> > You can use akka testkit
>> >
>> > Example:
>> >
>> >
>> >
>> https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
>> >
>> > --
>> > Nan Zhu
>> >
>> > On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
>> >
>> > Hi everyone,
>> >
>> > I’m adding some new message passing between the Master and Worker
>> actors in
>> > order to address https://issues.apache.org/jira/browse/SPARK-3736 .
>> >
>> > I was wondering if these kinds of interactions are tested in the
>> automated
>> > Jenkins test suite, and if so, where I could find some examples to help
>> me
>> > do the same.
>> >
>> > Thanks!
>> >
>> > -Matt Cheah
>> >
>> >
>> >
>> >
>> >
>>
>
>
Re: Unit testing Master-Worker Message Passing
Posted by Chester Chen <ch...@alpinenow.com>.
You can call resolve method on ActorSelection.resolveOne() to see if the
actor is still there or the path is correct. The method returns a future
and you can wait for it with timeout. This way, you know the actor is live
or already dead or incorrect.
Another way, is to send Identify method to ActorSystem, if it returns with
correct identified message; then you can act on it, otherwise, ...
hope this helps
Chester
On Wed, Oct 15, 2014 at 1:38 PM, Matthew Cheah <ma...@gmail.com>
wrote:
> What's happening when I do this is that the Worker tries to get the Master
> actor by calling context.actorSelection(), and the RegisterWorker message
> gets sent to the dead letters mailbox instead of being picked up by
> expectMsg. I'm new to Akka and I've tried various ways to registering a
> "mock" master to no avail.
>
> I would think there would be at least some kind of test for master - worker
> message passing, no?
>
> On Wed, Oct 15, 2014 at 11:28 AM, Nan Zhu <zh...@gmail.com> wrote:
>
> > I don’t think there are test cases for Worker itself
> >
> >
> > You can
> >
> >
> > val actorRef = TestActorRef[Master](Props(classOf[Master], ...))(
> > actorSystem) actorRef.underlyingActor.receive(Heartbeat)
> >
> > and use expectMsg to test if Master can reply correct message by
> assuming
> > Worker is absolutely correct
> >
> > Then in another test case to test if Worker can send register message to
> > Master after receiving Master’s “re-register” instruction, (in this test
> > case assuming that the Master is absolutely right)
> >
> > Best,
> >
> > --
> > Nan Zhu
> >
> > On Wednesday, October 15, 2014 at 2:04 PM, Matthew Cheah wrote:
> >
> > Thanks, the example was helpful.
> >
> > However, testing the Worker itself is a lot more complicated than
> > WorkerWatcher, since the Worker class is quite a bit more complex. Are
> > there any tests that inspect the Worker itself?
> >
> > Thanks,
> >
> > -Matt Cheah
> >
> > On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu <zh...@gmail.com> wrote:
> >
> > You can use akka testkit
> >
> > Example:
> >
> >
> >
> https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
> >
> > --
> > Nan Zhu
> >
> > On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
> >
> > Hi everyone,
> >
> > I’m adding some new message passing between the Master and Worker actors
> in
> > order to address https://issues.apache.org/jira/browse/SPARK-3736 .
> >
> > I was wondering if these kinds of interactions are tested in the
> automated
> > Jenkins test suite, and if so, where I could find some examples to help
> me
> > do the same.
> >
> > Thanks!
> >
> > -Matt Cheah
> >
> >
> >
> >
> >
>
Re: Unit testing Master-Worker Message Passing
Posted by Matthew Cheah <ma...@gmail.com>.
What's happening when I do this is that the Worker tries to get the Master
actor by calling context.actorSelection(), and the RegisterWorker message
gets sent to the dead letters mailbox instead of being picked up by
expectMsg. I'm new to Akka and I've tried various ways to registering a
"mock" master to no avail.
I would think there would be at least some kind of test for master - worker
message passing, no?
On Wed, Oct 15, 2014 at 11:28 AM, Nan Zhu <zh...@gmail.com> wrote:
> I don’t think there are test cases for Worker itself
>
>
> You can
>
>
> val actorRef = TestActorRef[Master](Props(classOf[Master], ...))(
> actorSystem) actorRef.underlyingActor.receive(Heartbeat)
>
> and use expectMsg to test if Master can reply correct message by assuming
> Worker is absolutely correct
>
> Then in another test case to test if Worker can send register message to
> Master after receiving Master’s “re-register” instruction, (in this test
> case assuming that the Master is absolutely right)
>
> Best,
>
> --
> Nan Zhu
>
> On Wednesday, October 15, 2014 at 2:04 PM, Matthew Cheah wrote:
>
> Thanks, the example was helpful.
>
> However, testing the Worker itself is a lot more complicated than
> WorkerWatcher, since the Worker class is quite a bit more complex. Are
> there any tests that inspect the Worker itself?
>
> Thanks,
>
> -Matt Cheah
>
> On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu <zh...@gmail.com> wrote:
>
> You can use akka testkit
>
> Example:
>
>
> https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
>
> --
> Nan Zhu
>
> On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
>
> Hi everyone,
>
> I’m adding some new message passing between the Master and Worker actors in
> order to address https://issues.apache.org/jira/browse/SPARK-3736 .
>
> I was wondering if these kinds of interactions are tested in the automated
> Jenkins test suite, and if so, where I could find some examples to help me
> do the same.
>
> Thanks!
>
> -Matt Cheah
>
>
>
>
>
Re: Unit testing Master-Worker Message Passing
Posted by Nan Zhu <zh...@gmail.com>.
I don’t think there are test cases for Worker itself
You can
val actorRef = TestActorRef[Master](Props(classOf[Master], ...))(actorSystem) actorRef.underlyingActor.receive(Heartbeat)
and use expectMsg to test if Master can reply correct message by assuming Worker is absolutely correct
Then in another test case to test if Worker can send register message to Master after receiving Master’s “re-register” instruction, (in this test case assuming that the Master is absolutely right)
Best,
--
Nan Zhu
On Wednesday, October 15, 2014 at 2:04 PM, Matthew Cheah wrote:
> Thanks, the example was helpful.
>
> However, testing the Worker itself is a lot more complicated than WorkerWatcher, since the Worker class is quite a bit more complex. Are there any tests that inspect the Worker itself?
>
> Thanks,
>
> -Matt Cheah
>
> On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu <zhunanmcgill@gmail.com (mailto:zhunanmcgill@gmail.com)> wrote:
> > You can use akka testkit
> >
> > Example:
> >
> > https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
> >
> > --
> > Nan Zhu
> >
> >
> > On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
> >
> > > Hi everyone,
> > >
> > > I’m adding some new message passing between the Master and Worker actors in
> > > order to address https://issues.apache.org/jira/browse/SPARK-3736 .
> > >
> > > I was wondering if these kinds of interactions are tested in the automated
> > > Jenkins test suite, and if so, where I could find some examples to help me
> > > do the same.
> > >
> > > Thanks!
> > >
> > > -Matt Cheah
> >
>
Re: Unit testing Master-Worker Message Passing
Posted by Matthew Cheah <ma...@gmail.com>.
Thanks, the example was helpful.
However, testing the Worker itself is a lot more complicated than
WorkerWatcher, since the Worker class is quite a bit more complex. Are
there any tests that inspect the Worker itself?
Thanks,
-Matt Cheah
On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu <zh...@gmail.com> wrote:
> You can use akka testkit
>
> Example:
>
>
> https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
>
> --
> Nan Zhu
>
> On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
>
> Hi everyone,
>
> I’m adding some new message passing between the Master and Worker actors in
> order to address https://issues.apache.org/jira/browse/SPARK-3736 .
>
> I was wondering if these kinds of interactions are tested in the automated
> Jenkins test suite, and if so, where I could find some examples to help me
> do the same.
>
> Thanks!
>
> -Matt Cheah
>
>
>
Re: Unit testing Master-Worker Message Passing
Posted by Nan Zhu <zh...@gmail.com>.
You can use akka testkit
Example:
https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
--
Nan Zhu
On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
> Hi everyone,
>
> I’m adding some new message passing between the Master and Worker actors in
> order to address https://issues.apache.org/jira/browse/SPARK-3736 .
>
> I was wondering if these kinds of interactions are tested in the automated
> Jenkins test suite, and if so, where I could find some examples to help me
> do the same.
>
> Thanks!
>
> -Matt Cheah