You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Corey Nolet <cj...@gmail.com> on 2014/08/05 02:49:08 UTC
Good way to test when topology in local cluster is "fully active"
I'm testing some sliding window algorithms with tuples emitted from a mock
spout based on a timer but the amount of time it takes the topology to
fully start up and activate seems to vary from computer to computer.
Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
tests are breaking because the time to activate the topology is taking
longer (because of Netty possibly?). I'd like to make my tests more
resilient to things like this.
Is there something I can look at in LocalCluster where I could do
"while(!notActive) { Thread.sleep(50) }" ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
Re: Good way to test when topology in local cluster is "fully active"
Posted by Vincent Russell <vi...@gmail.com>.
No worries. I haven't found the testing framework the easiest thing to
use. If you see errors that say something like "cannot convert object to
iterable" I use the code below to disable storm from being able to call
System.exit()
private static class ExitTrappedException extends SecurityException {
}
private static void forbidSystemExitCall() {
final SecurityManager securityManager = new SecurityManager() {
public void checkPermission(Permission permission) {
if (permission.getName().startsWith("exitVM")) {
throw new ExitTrappedException();
}
}
};
System.setSecurityManager(securityManager);
}
private static void enableSystemExitCall() {
System.setSecurityManager(null);
}
On Mon, Aug 4, 2014 at 11:02 PM, Corey Nolet <cj...@gmail.com> wrote:
> Nevermind, I wrote that before looking. This has been around since 0.8.1.
> Thanks again Vincent!
>
>
> On Mon, Aug 4, 2014 at 11:01 PM, Corey Nolet <cj...@gmail.com> wrote:
>
>> Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much
>> into what's changed yet, other than Netty.
>>
>>
>> On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell <
>> vincent.russell@gmail.com> wrote:
>>
>>> Corey,
>>>
>>> Have you tried using the integration testing framework that comes with
>>> storm?
>>>
>>>
>>> Testing.withSimulatedTimeLocalCluster(mkClusterParam,
>>> new TestJob() {
>>> @Override
>>> public void run(ILocalCluster cluster) throws Exception {
>>>
>>> CompleteTopologyParam completeTopologyParam = new
>>> CompleteTopologyParam();
>>> completeTopologyParam
>>> .setMockedSources(mockedSources);
>>> completeTopologyParam.setStormConf(daemonConf);
>>>
>>> completeTopologyParam.setTopologyName(getTopologyName());
>>> Map result = Testing.completeTopology(cluster,
>>> topology, completeTopologyParam);
>>>
>>> });
>>>
>>> -Vincent
>>>
>>> On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet <cj...@gmail.com> wrote:
>>>
>>>> I'm testing some sliding window algorithms with tuples emitted from a
>>>> mock spout based on a timer but the amount of time it takes the topology to
>>>> fully start up and activate seems to vary from computer to computer.
>>>> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
>>>> tests are breaking because the time to activate the topology is taking
>>>> longer (because of Netty possibly?). I'd like to make my tests more
>>>> resilient to things like this.
>>>>
>>>> Is there something I can look at in LocalCluster where I could do
>>>> "while(!notActive) { Thread.sleep(50) }" ?
>>>>
>>>> This is what my test looks like currently:
>>>>
>>>> StormTopology topology = buildTopology(...);
>>>> Config conf = new Config();
>>>> conf.setNumWorkers(1);
>>>>
>>>> LocalCluster cluster = new LocalCluster();
>>>> cluster.submitTopology(getTopologyName(), conf, topology);
>>>>
>>>> try {
>>>> Thread.sleep(4000);
>>>> } catch (InterruptedException e) {
>>>> e.printStackTrace();
>>>> }
>>>>
>>>> cluster.shutdown();
>>>>
>>>> assertEquals(4, MockSinkBolt.getEvents().size());
>>>>
>>>>
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>
>>
>
Re: Good way to test when topology in local cluster is "fully active"
Posted by Corey Nolet <cj...@gmail.com>.
Nevermind, I wrote that before looking. This has been around since 0.8.1.
Thanks again Vincent!
On Mon, Aug 4, 2014 at 11:01 PM, Corey Nolet <cj...@gmail.com> wrote:
> Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much
> into what's changed yet, other than Netty.
>
>
> On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell <
> vincent.russell@gmail.com> wrote:
>
>> Corey,
>>
>> Have you tried using the integration testing framework that comes with
>> storm?
>>
>>
>> Testing.withSimulatedTimeLocalCluster(mkClusterParam,
>> new TestJob() {
>> @Override
>> public void run(ILocalCluster cluster) throws Exception {
>>
>> CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam();
>> completeTopologyParam
>> .setMockedSources(mockedSources);
>> completeTopologyParam.setStormConf(daemonConf);
>>
>> completeTopologyParam.setTopologyName(getTopologyName());
>> Map result = Testing.completeTopology(cluster,
>> topology, completeTopologyParam);
>>
>> });
>>
>> -Vincent
>>
>> On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet <cj...@gmail.com> wrote:
>>
>>> I'm testing some sliding window algorithms with tuples emitted from a
>>> mock spout based on a timer but the amount of time it takes the topology to
>>> fully start up and activate seems to vary from computer to computer.
>>> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
>>> tests are breaking because the time to activate the topology is taking
>>> longer (because of Netty possibly?). I'd like to make my tests more
>>> resilient to things like this.
>>>
>>> Is there something I can look at in LocalCluster where I could do
>>> "while(!notActive) { Thread.sleep(50) }" ?
>>>
>>> This is what my test looks like currently:
>>>
>>> StormTopology topology = buildTopology(...);
>>> Config conf = new Config();
>>> conf.setNumWorkers(1);
>>>
>>> LocalCluster cluster = new LocalCluster();
>>> cluster.submitTopology(getTopologyName(), conf, topology);
>>>
>>> try {
>>> Thread.sleep(4000);
>>> } catch (InterruptedException e) {
>>> e.printStackTrace();
>>> }
>>>
>>> cluster.shutdown();
>>>
>>> assertEquals(4, MockSinkBolt.getEvents().size());
>>>
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>
>
Re: Good way to test when topology in local cluster is "fully active"
Posted by Corey Nolet <cj...@gmail.com>.
Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much into
what's changed yet, other than Netty.
On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell <vi...@gmail.com>
wrote:
> Corey,
>
> Have you tried using the integration testing framework that comes with
> storm?
>
>
> Testing.withSimulatedTimeLocalCluster(mkClusterParam,
> new TestJob() {
> @Override
> public void run(ILocalCluster cluster) throws Exception {
>
> CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam();
> completeTopologyParam
> .setMockedSources(mockedSources);
> completeTopologyParam.setStormConf(daemonConf);
>
> completeTopologyParam.setTopologyName(getTopologyName());
> Map result = Testing.completeTopology(cluster,
> topology, completeTopologyParam);
>
> });
>
> -Vincent
>
> On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet <cj...@gmail.com> wrote:
>
>> I'm testing some sliding window algorithms with tuples emitted from a
>> mock spout based on a timer but the amount of time it takes the topology to
>> fully start up and activate seems to vary from computer to computer.
>> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
>> tests are breaking because the time to activate the topology is taking
>> longer (because of Netty possibly?). I'd like to make my tests more
>> resilient to things like this.
>>
>> Is there something I can look at in LocalCluster where I could do
>> "while(!notActive) { Thread.sleep(50) }" ?
>>
>> This is what my test looks like currently:
>>
>> StormTopology topology = buildTopology(...);
>> Config conf = new Config();
>> conf.setNumWorkers(1);
>>
>> LocalCluster cluster = new LocalCluster();
>> cluster.submitTopology(getTopologyName(), conf, topology);
>>
>> try {
>> Thread.sleep(4000);
>> } catch (InterruptedException e) {
>> e.printStackTrace();
>> }
>>
>> cluster.shutdown();
>>
>> assertEquals(4, MockSinkBolt.getEvents().size());
>>
>>
>>
>> Thanks!
>>
>>
>>
>
Re: Good way to test when topology in local cluster is "fully active"
Posted by Vincent Russell <vi...@gmail.com>.
Corey,
Have you tried using the integration testing framework that comes with
storm?
Testing.withSimulatedTimeLocalCluster(mkClusterParam,
new TestJob() {
@Override
public void run(ILocalCluster cluster) throws Exception {
CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam();
completeTopologyParam
.setMockedSources(mockedSources);
completeTopologyParam.setStormConf(daemonConf);
completeTopologyParam.setTopologyName(getTopologyName());
Map result = Testing.completeTopology(cluster,
topology, completeTopologyParam);
});
-Vincent
On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet <cj...@gmail.com> wrote:
> I'm testing some sliding window algorithms with tuples emitted from a mock
> spout based on a timer but the amount of time it takes the topology to
> fully start up and activate seems to vary from computer to computer.
> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
> tests are breaking because the time to activate the topology is taking
> longer (because of Netty possibly?). I'd like to make my tests more
> resilient to things like this.
>
> Is there something I can look at in LocalCluster where I could do
> "while(!notActive) { Thread.sleep(50) }" ?
>
> This is what my test looks like currently:
>
> StormTopology topology = buildTopology(...);
> Config conf = new Config();
> conf.setNumWorkers(1);
>
> LocalCluster cluster = new LocalCluster();
> cluster.submitTopology(getTopologyName(), conf, topology);
>
> try {
> Thread.sleep(4000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
>
> cluster.shutdown();
>
> assertEquals(4, MockSinkBolt.getEvents().size());
>
>
>
> Thanks!
>
>
>
Re: Good way to test when topology in local cluster is "fully active"
Posted by Corey Nolet <cj...@gmail.com>.
Vincent & P.Taylor,
I played with the testing framework for a little bit last night and don't
see any easy way to provide pauses in between the emissions of the mock
tuples. For instance, my sliding window semantics are heavily orchestrated
by time evictions and triggers which mean that I need to be able to time
the tuples being fed into the tests (i.e. emit a tuple every 500ms and run
the test for 25 secons).
On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> My guess is that the slowdown you are seeing is a result of the new
> version of ZooKeeper and how it handles IPv4/6.
>
> Try adding the following JVM parameter when running your tests:
>
> -Djava.net.preferIPv4Stack=true
>
> -Taylor
>
> On Aug 4, 2014, at 8:49 PM, Corey Nolet <cj...@gmail.com> wrote:
>
> > I'm testing some sliding window algorithms with tuples emitted from a
> mock spout based on a timer but the amount of time it takes the topology to
> fully start up and activate seems to vary from computer to computer.
> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
> tests are breaking because the time to activate the topology is taking
> longer (because of Netty possibly?). I'd like to make my tests more
> resilient to things like this.
> >
> > Is there something I can look at in LocalCluster where I could do
> "while(!notActive) { Thread.sleep(50) }" ?
> >
> > This is what my test looks like currently:
> >
> > StormTopology topology = buildTopology(...);
> > Config conf = new Config();
> > conf.setNumWorkers(1);
> >
> > LocalCluster cluster = new LocalCluster();
> > cluster.submitTopology(getTopologyName(), conf, topology);
> >
> > try {
> > Thread.sleep(4000);
> > } catch (InterruptedException e) {
> > e.printStackTrace();
> > }
> >
> > cluster.shutdown();
> >
> > assertEquals(4, MockSinkBolt.getEvents().size());
> >
> >
> >
> > Thanks!
> >
> >
>
>
Re: Good way to test when topology in local cluster is "fully active"
Posted by Corey Nolet <cj...@gmail.com>.
Sorry- the ipv4 fix worked.
On Tue, Aug 5, 2014 at 9:13 PM, Corey Nolet <cj...@gmail.com> wrote:
> This did work. Thanks!
>
>
> On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
>
>> My guess is that the slowdown you are seeing is a result of the new
>> version of ZooKeeper and how it handles IPv4/6.
>>
>> Try adding the following JVM parameter when running your tests:
>>
>> -Djava.net.preferIPv4Stack=true
>>
>> -Taylor
>>
>> On Aug 4, 2014, at 8:49 PM, Corey Nolet <cj...@gmail.com> wrote:
>>
>> > I'm testing some sliding window algorithms with tuples emitted from a
>> mock spout based on a timer but the amount of time it takes the topology to
>> fully start up and activate seems to vary from computer to computer.
>> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
>> tests are breaking because the time to activate the topology is taking
>> longer (because of Netty possibly?). I'd like to make my tests more
>> resilient to things like this.
>> >
>> > Is there something I can look at in LocalCluster where I could do
>> "while(!notActive) { Thread.sleep(50) }" ?
>> >
>> > This is what my test looks like currently:
>> >
>> > StormTopology topology = buildTopology(...);
>> > Config conf = new Config();
>> > conf.setNumWorkers(1);
>> >
>> > LocalCluster cluster = new LocalCluster();
>> > cluster.submitTopology(getTopologyName(), conf, topology);
>> >
>> > try {
>> > Thread.sleep(4000);
>> > } catch (InterruptedException e) {
>> > e.printStackTrace();
>> > }
>> >
>> > cluster.shutdown();
>> >
>> > assertEquals(4, MockSinkBolt.getEvents().size());
>> >
>> >
>> >
>> > Thanks!
>> >
>> >
>>
>>
>
Re: Good way to test when topology in local cluster is "fully active"
Posted by Corey Nolet <cj...@gmail.com>.
This did work. Thanks!
On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> My guess is that the slowdown you are seeing is a result of the new
> version of ZooKeeper and how it handles IPv4/6.
>
> Try adding the following JVM parameter when running your tests:
>
> -Djava.net.preferIPv4Stack=true
>
> -Taylor
>
> On Aug 4, 2014, at 8:49 PM, Corey Nolet <cj...@gmail.com> wrote:
>
> > I'm testing some sliding window algorithms with tuples emitted from a
> mock spout based on a timer but the amount of time it takes the topology to
> fully start up and activate seems to vary from computer to computer.
> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
> tests are breaking because the time to activate the topology is taking
> longer (because of Netty possibly?). I'd like to make my tests more
> resilient to things like this.
> >
> > Is there something I can look at in LocalCluster where I could do
> "while(!notActive) { Thread.sleep(50) }" ?
> >
> > This is what my test looks like currently:
> >
> > StormTopology topology = buildTopology(...);
> > Config conf = new Config();
> > conf.setNumWorkers(1);
> >
> > LocalCluster cluster = new LocalCluster();
> > cluster.submitTopology(getTopologyName(), conf, topology);
> >
> > try {
> > Thread.sleep(4000);
> > } catch (InterruptedException e) {
> > e.printStackTrace();
> > }
> >
> > cluster.shutdown();
> >
> > assertEquals(4, MockSinkBolt.getEvents().size());
> >
> >
> >
> > Thanks!
> >
> >
>
>
Re: Good way to test when topology in local cluster is "fully active"
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
My guess is that the slowdown you are seeing is a result of the new version of ZooKeeper and how it handles IPv4/6.
Try adding the following JVM parameter when running your tests:
-Djava.net.preferIPv4Stack=true
-Taylor
On Aug 4, 2014, at 8:49 PM, Corey Nolet <cj...@gmail.com> wrote:
> I'm testing some sliding window algorithms with tuples emitted from a mock spout based on a timer but the amount of time it takes the topology to fully start up and activate seems to vary from computer to computer. Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my tests are breaking because the time to activate the topology is taking longer (because of Netty possibly?). I'd like to make my tests more resilient to things like this.
>
> Is there something I can look at in LocalCluster where I could do "while(!notActive) { Thread.sleep(50) }" ?
>
> This is what my test looks like currently:
>
> StormTopology topology = buildTopology(...);
> Config conf = new Config();
> conf.setNumWorkers(1);
>
> LocalCluster cluster = new LocalCluster();
> cluster.submitTopology(getTopologyName(), conf, topology);
>
> try {
> Thread.sleep(4000);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
>
> cluster.shutdown();
>
> assertEquals(4, MockSinkBolt.getEvents().size());
>
>
>
> Thanks!
>
>