You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Harsha <st...@harsha.io> on 2015/11/05 02:20:51 UTC
[Discusson] Storm System Tests
Hi All,
As community we are growing and adding new and exciting
features to Storm and also we've ever growing connector which
only helps in storm adoption. One thing we've severely lacking
is system tests. There are unit tests which acts as
integration tests to storm-core but there are no system wide
integration tests that can spin up kafka nodes and storm ,
hbase and run a topology that can make sure the data is
getting into hbase.
We at Hortonworks use our test topologies run some of these
tests but this integration code to spin up vms or nodes is hard
to share. In apache kafka we are using ducktape to write system
tests so far its working out good. If there are any other
frameworks you've in mind we can definitely take a look. But as
a community we need start looking at
https://github.com/confluentinc/ducktape or similar frameworks
to start writing systems tests. Ducktape makes it easy to run
in a vm or any other infrastructure. Appreciate any feedback
on this.
Thanks,
Harsha
Re: [Discusson] Storm System Tests
Posted by Derek Dagit <de...@yahoo-inc.com.INVALID>.
> If we agree that it is something we want to implement here, I will create a JIRA ticket for this and get it done
+1
--
Derek
----- Original Message -----
From: Hugo Da Cruz Louro <hl...@hortonworks.com>
To: "dev@storm.apache.org" <de...@storm.apache.org>; Bobby Evans <ev...@yahoo-inc.com>
Cc:
Sent: Thursday, November 5, 2015 1:37 PM
Subject: Re: [Discusson] Storm System Tests
I a agree with these three levels of testing, and that at the very least we should keep unit tests separated from the system/integration tests. One huge advantage would be to quickly run unit tests that hopefully are more predictable, and thus avoid intermittent test fails.
Bobby just mentioned but I had already in mind that it would be useful to create different maven profiles for integration and unit tests. I have done something similar in a different project and it works really well. If we agree that it is something we want to implement here, I will create a JIRA ticket for this and get it done. Please let me know.
Thanks,
Hugo
> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
> I personally would like to see three levels of testing.
> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
> These would be run frequently but not necessarily a part of CI initially.
> - Bobby
>
>
> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io> wrote:
>
>
> Hi All,
> As community we are growing and adding new and exciting
> features to Storm and also we've ever growing connector which
> only helps in storm adoption. One thing we've severely lacking
> is system tests. There are unit tests which acts as
> integration tests to storm-core but there are no system wide
> integration tests that can spin up kafka nodes and storm ,
> hbase and run a topology that can make sure the data is
> getting into hbase.
> We at Hortonworks use our test topologies run some of these
> tests but this integration code to spin up vms or nodes is hard
> to share. In apache kafka we are using ducktape to write system
> tests so far its working out good. If there are any other
> frameworks you've in mind we can definitely take a look. But as
> a community we need start looking at
> https://github.com/confluentinc/ducktape or similar frameworks
> to start writing systems tests. Ducktape makes it easy to run
> in a vm or any other infrastructure. Appreciate any feedback
> on this.
>
> Thanks,
> Harsha
>
>
Re: [Discusson] Storm System Tests
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Additional relevant discussion happening here as well:
https://issues.apache.org/jira/browse/STORM-1179 <https://issues.apache.org/jira/browse/STORM-1179>
I also have my own set of utilities that I use for testing fault tolerance, etc. — the main feature is automated execution of commands over ssh and simulation of network partitions, etc. with ipchains.
If duck tape supports something similar, that would be great, especially when it comes to testing things like Nimbus HA.
-Taylor
> On Nov 6, 2015, at 9:40 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> Yes we too have similar tests based on internal Yahoo testing. Really it comes down to a few operations that the test framework needs to be able to do 1) node/VM allocation 2) deployment 3) modify configuration/processes on running nodes 4) run actual tests.
> The first three are often specific to the setup. Is it open stack? Amazon EC2? Are we using ssh to go between nodes, or is it something else? All of this is something that we should be able to accomplish with an abstraction layer. ducktape appears to have a fairly decent abstraction layer for the first even allowing for a static JSON config for the hosts. The only other requirement seems to be ssh access between the nodes so it can run various python scripts. As we are developing these tests we probably want to be sure that we have a pluggable abstraction for the other layers as well. That way if I am using RPMs and Chef to do a deployment and be sure my RPM packaging is correct I can do that, and if someone else just wants to take the TGZ and scp it everywhere that works too.
> I also hope that we can limit the score of these tests. The current Yahoo integration tests take several hours to run because we are constantly trying to reconfigure the cluster to test different setups. Ideally we can test failure scenarios and basic integration with other tools, but anything that can be done on a LocalCluster should probably be tested there, to reduce the total running time and debugging complexity.
> - Bobby
>
> On Thursday, November 5, 2015 10:51 PM, Harsha <st...@harsha.io> wrote:
>
>
> Thanks Longda. Can you update this JIRA with list of tests cases that
> you are running in this JIRA
> https://issues.apache.org/jira/browse/STORM-1179
> -Harsha
>
> On Thu, Nov 5, 2015, at 08:32 PM, 封仲淹(纪君祥LongdaFeng) wrote:
>> +1
>> By the way, Tong(王桐) can work for this. Tong is responsible for the
>> automatically daily JStorm test, the only problem is that the
>> automatically daily JStorm test is basing on Alibaba testing framework.
>> Some of the components can't run on outside.
>> But we can share the test cases firstly. Then we will try to port the old
>> test cases to the storm tests.
>>
>> regardsLongda
>> ------------------------------------------------------------------From:P.
>> Taylor Goetz <pt...@gmail.com>Send Time:2015年11月6日(星期五)
>> 12:12To:dev@storm.apache.org <de...@storm.apache.org>Subject:Re:
>> [Discusson] Storm System Tests
>> +1
>> I think we should experiment with this. If it can serve us well, and we
>> all agree, then we should consider adopting it.
>>
>> From what I see:
>>
>> * Icense is compatible
>> * fairly well documented.
>>
>> What potentially worries me:
>>
>> * Maintenance. Can we count on updates? Is the ducktape community open to
>> accepting patches from us? If so, what would be the turnaround time for
>> acceptance?
>>
>> -Taylor
>>
>>> On Nov 5, 2015, at 8:54 PM, Harsha <st...@harsha.io> wrote:
>>>
>>> The reason for suggesting ducktape not just used in apache kafka but
>>> also its getting security services integration like kdc and already has
>>> zookeeper. This framework can work with vagrant vms or amazon ec2 etc..
>>>
>>>> On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
>>>> Great, will make this a priority. Created a
>>>> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
>>>> assigned it to me.
>>>>
>>>> https://issues.apache.org/jira/browse/STORM-1179
>>>>
>>>>
>>>> On Nov 5, 2015, at 11:40 AM, Bobby Evans
>>>> <ev...@yahoo-inc.com>> wrote:
>>>>
>>>> Hugo,
>>>>
>>>> I would love to see that happen. It has been on my list for a while, but
>>>> I have never found the time to do it. I personally am +1 on this,
>>>> hopefully we can do this quickly in preparation for a 0.11.0 release.
>>>>
>>>> - Bobby
>>>>
>>>>
>>>>
>>>> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
>>>> <hl...@hortonworks.com>> wrote:
>>>>
>>>>
>>>> I a agree with these three levels of testing, and that at the very least
>>>> we should keep unit tests separated from the system/integration tests.
>>>> One huge advantage would be to quickly run unit tests that hopefully are
>>>> more predictable, and thus avoid intermittent test fails.
>>>>
>>>> Bobby just mentioned but I had already in mind that it would be useful to
>>>> create different maven profiles for integration and unit tests. I have
>>>> done something similar in a different project and it works really well.
>>>> If we agree that it is something we want to implement here, I will create
>>>> a JIRA ticket for this and get it done. Please let me know.
>>>>
>>>> Thanks,
>>>> Hugo
>>>>
>>>>> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
>>>>>
>>>>> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
>>>>> I personally would like to see three levels of testing.
>>>>> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
>>>>> These would be run frequently but not necessarily a part of CI initially.
>>>>> - Bobby
>>>>>
>>>>>
>>>>> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io>> wrote:
>>>>>
>>>>>
>>>>> Hi All,
>>>>> As community we are growing and adding new and exciting
>>>>> features to Storm and also we've ever growing connector which
>>>>> only helps in storm adoption. One thing we've severely lacking
>>>>> is system tests. There are unit tests which acts as
>>>>> integration tests to storm-core but there are no system wide
>>>>> integration tests that can spin up kafka nodes and storm ,
>>>>> hbase and run a topology that can make sure the data is
>>>>> getting into hbase.
>>>>> We at Hortonworks use our test topologies run some of these
>>>>> tests but this integration code to spin up vms or nodes is hard
>>>>> to share. In apache kafka we are using ducktape to write system
>>>>> tests so far its working out good. If there are any other
>>>>> frameworks you've in mind we can definitely take a look. But as
>>>>> a community we need start looking at
>>>>> https://github.com/confluentinc/ducktape or similar frameworks
>>>>> to start writing systems tests. Ducktape makes it easy to run
>>>>> in a vm or any other infrastructure. Appreciate any feedback
>>>>> on this.
>>>>>
>>>>> Thanks,
>>>>> Harsha
>>>>
>>>>
>>>>
>
Re: [Discusson] Storm System Tests
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Yes we too have similar tests based on internal Yahoo testing. Really it comes down to a few operations that the test framework needs to be able to do 1) node/VM allocation 2) deployment 3) modify configuration/processes on running nodes 4) run actual tests.
The first three are often specific to the setup. Is it open stack? Amazon EC2? Are we using ssh to go between nodes, or is it something else? All of this is something that we should be able to accomplish with an abstraction layer. ducktape appears to have a fairly decent abstraction layer for the first even allowing for a static JSON config for the hosts. The only other requirement seems to be ssh access between the nodes so it can run various python scripts. As we are developing these tests we probably want to be sure that we have a pluggable abstraction for the other layers as well. That way if I am using RPMs and Chef to do a deployment and be sure my RPM packaging is correct I can do that, and if someone else just wants to take the TGZ and scp it everywhere that works too.
I also hope that we can limit the score of these tests. The current Yahoo integration tests take several hours to run because we are constantly trying to reconfigure the cluster to test different setups. Ideally we can test failure scenarios and basic integration with other tools, but anything that can be done on a LocalCluster should probably be tested there, to reduce the total running time and debugging complexity.
- Bobby
On Thursday, November 5, 2015 10:51 PM, Harsha <st...@harsha.io> wrote:
Thanks Longda. Can you update this JIRA with list of tests cases that
you are running in this JIRA
https://issues.apache.org/jira/browse/STORM-1179
-Harsha
On Thu, Nov 5, 2015, at 08:32 PM, 封仲淹(纪君祥LongdaFeng) wrote:
> +1
> By the way, Tong(王桐) can work for this. Tong is responsible for the
> automatically daily JStorm test, the only problem is that the
> automatically daily JStorm test is basing on Alibaba testing framework.
> Some of the components can't run on outside.
> But we can share the test cases firstly. Then we will try to port the old
> test cases to the storm tests.
>
> regardsLongda
> ------------------------------------------------------------------From:P.
> Taylor Goetz <pt...@gmail.com>Send Time:2015年11月6日(星期五)
> 12:12To:dev@storm.apache.org <de...@storm.apache.org>Subject:Re:
> [Discusson] Storm System Tests
> +1
> I think we should experiment with this. If it can serve us well, and we
> all agree, then we should consider adopting it.
>
> From what I see:
>
> * Icense is compatible
> * fairly well documented.
>
> What potentially worries me:
>
> * Maintenance. Can we count on updates? Is the ducktape community open to
> accepting patches from us? If so, what would be the turnaround time for
> acceptance?
>
> -Taylor
>
> > On Nov 5, 2015, at 8:54 PM, Harsha <st...@harsha.io> wrote:
> >
> > The reason for suggesting ducktape not just used in apache kafka but
> > also its getting security services integration like kdc and already has
> > zookeeper. This framework can work with vagrant vms or amazon ec2 etc..
> >
> >> On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
> >> Great, will make this a priority. Created a
> >> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
> >> assigned it to me.
> >>
> >> https://issues.apache.org/jira/browse/STORM-1179
> >>
> >>
> >> On Nov 5, 2015, at 11:40 AM, Bobby Evans
> >> <ev...@yahoo-inc.com>> wrote:
> >>
> >> Hugo,
> >>
> >> I would love to see that happen. It has been on my list for a while, but
> >> I have never found the time to do it. I personally am +1 on this,
> >> hopefully we can do this quickly in preparation for a 0.11.0 release.
> >>
> >> - Bobby
> >>
> >>
> >>
> >> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
> >> <hl...@hortonworks.com>> wrote:
> >>
> >>
> >> I a agree with these three levels of testing, and that at the very least
> >> we should keep unit tests separated from the system/integration tests.
> >> One huge advantage would be to quickly run unit tests that hopefully are
> >> more predictable, and thus avoid intermittent test fails.
> >>
> >> Bobby just mentioned but I had already in mind that it would be useful to
> >> create different maven profiles for integration and unit tests. I have
> >> done something similar in a different project and it works really well.
> >> If we agree that it is something we want to implement here, I will create
> >> a JIRA ticket for this and get it done. Please let me know.
> >>
> >> Thanks,
> >> Hugo
> >>
> >>> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
> >>>
> >>> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
> >>> I personally would like to see three levels of testing.
> >>> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
> >>> These would be run frequently but not necessarily a part of CI initially.
> >>> - Bobby
> >>>
> >>>
> >>> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io>> wrote:
> >>>
> >>>
> >>> Hi All,
> >>> As community we are growing and adding new and exciting
> >>> features to Storm and also we've ever growing connector which
> >>> only helps in storm adoption. One thing we've severely lacking
> >>> is system tests. There are unit tests which acts as
> >>> integration tests to storm-core but there are no system wide
> >>> integration tests that can spin up kafka nodes and storm ,
> >>> hbase and run a topology that can make sure the data is
> >>> getting into hbase.
> >>> We at Hortonworks use our test topologies run some of these
> >>> tests but this integration code to spin up vms or nodes is hard
> >>> to share. In apache kafka we are using ducktape to write system
> >>> tests so far its working out good. If there are any other
> >>> frameworks you've in mind we can definitely take a look. But as
> >>> a community we need start looking at
> >>> https://github.com/confluentinc/ducktape or similar frameworks
> >>> to start writing systems tests. Ducktape makes it easy to run
> >>> in a vm or any other infrastructure. Appreciate any feedback
> >>> on this.
> >>>
> >>> Thanks,
> >>> Harsha
> >>
> >>
> >>
Re: [Discusson] Storm System Tests
Posted by Paul Poulosky <pp...@yahoo-inc.com.INVALID>.
I would like to work on this as well. We have some integration tests at Yahoo that we run regularly, but also depend on an internal testing framework, and also depend on internal deployment scripts, etc.
It would be good to get a set of true integration tests set up that can run in community. We could then reduce the number of tests we run internally to truly test Yahoo specific integration needs.
On Thursday, November 5, 2015 10:51 PM, Harsha <st...@harsha.io> wrote:
Thanks Longda. Can you update this JIRA with list of tests cases that
you are running in this JIRA
https://issues.apache.org/jira/browse/STORM-1179
-Harsha
On Thu, Nov 5, 2015, at 08:32 PM, 封仲淹(纪君祥LongdaFeng) wrote:
> +1
> By the way, Tong(王桐) can work for this. Tong is responsible for the
> automatically daily JStorm test, the only problem is that the
> automatically daily JStorm test is basing on Alibaba testing framework.
> Some of the components can't run on outside.
> But we can share the test cases firstly. Then we will try to port the old
> test cases to the storm tests.
>
> regardsLongda
> ------------------------------------------------------------------From:P.
> Taylor Goetz <pt...@gmail.com>Send Time:2015年11月6日(星期五)
> 12:12To:dev@storm.apache.org <de...@storm.apache.org>Subject:Re:
> [Discusson] Storm System Tests
> +1
> I think we should experiment with this. If it can serve us well, and we
> all agree, then we should consider adopting it.
>
> From what I see:
>
> * Icense is compatible
> * fairly well documented.
>
> What potentially worries me:
>
> * Maintenance. Can we count on updates? Is the ducktape community open to
> accepting patches from us? If so, what would be the turnaround time for
> acceptance?
>
> -Taylor
>
> > On Nov 5, 2015, at 8:54 PM, Harsha <st...@harsha.io> wrote:
> >
> > The reason for suggesting ducktape not just used in apache kafka but
> > also its getting security services integration like kdc and already has
> > zookeeper. This framework can work with vagrant vms or amazon ec2 etc..
> >
> >> On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
> >> Great, will make this a priority. Created a
> >> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
> >> assigned it to me.
> >>
> >> https://issues.apache.org/jira/browse/STORM-1179
> >>
> >>
> >> On Nov 5, 2015, at 11:40 AM, Bobby Evans
> >> <ev...@yahoo-inc.com>> wrote:
> >>
> >> Hugo,
> >>
> >> I would love to see that happen. It has been on my list for a while, but
> >> I have never found the time to do it. I personally am +1 on this,
> >> hopefully we can do this quickly in preparation for a 0.11.0 release.
> >>
> >> - Bobby
> >>
> >>
> >>
> >> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
> >> <hl...@hortonworks.com>> wrote:
> >>
> >>
> >> I a agree with these three levels of testing, and that at the very least
> >> we should keep unit tests separated from the system/integration tests.
> >> One huge advantage would be to quickly run unit tests that hopefully are
> >> more predictable, and thus avoid intermittent test fails.
> >>
> >> Bobby just mentioned but I had already in mind that it would be useful to
> >> create different maven profiles for integration and unit tests. I have
> >> done something similar in a different project and it works really well.
> >> If we agree that it is something we want to implement here, I will create
> >> a JIRA ticket for this and get it done. Please let me know.
> >>
> >> Thanks,
> >> Hugo
> >>
> >>> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
> >>>
> >>> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
> >>> I personally would like to see three levels of testing.
> >>> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
> >>> These would be run frequently but not necessarily a part of CI initially.
> >>> - Bobby
> >>>
> >>>
> >>> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io>> wrote:
> >>>
> >>>
> >>> Hi All,
> >>> As community we are growing and adding new and exciting
> >>> features to Storm and also we've ever growing connector which
> >>> only helps in storm adoption. One thing we've severely lacking
> >>> is system tests. There are unit tests which acts as
> >>> integration tests to storm-core but there are no system wide
> >>> integration tests that can spin up kafka nodes and storm ,
> >>> hbase and run a topology that can make sure the data is
> >>> getting into hbase.
> >>> We at Hortonworks use our test topologies run some of these
> >>> tests but this integration code to spin up vms or nodes is hard
> >>> to share. In apache kafka we are using ducktape to write system
> >>> tests so far its working out good. If there are any other
> >>> frameworks you've in mind we can definitely take a look. But as
> >>> a community we need start looking at
> >>> https://github.com/confluentinc/ducktape or similar frameworks
> >>> to start writing systems tests. Ducktape makes it easy to run
> >>> in a vm or any other infrastructure. Appreciate any feedback
> >>> on this.
> >>>
> >>> Thanks,
> >>> Harsha
> >>
> >>
> >>
Re: [Discusson] Storm System Tests
Posted by Harsha <st...@harsha.io>.
Thanks Longda. Can you update this JIRA with list of tests cases that
you are running in this JIRA
https://issues.apache.org/jira/browse/STORM-1179
-Harsha
On Thu, Nov 5, 2015, at 08:32 PM, 封仲淹(纪君祥LongdaFeng) wrote:
> +1
> By the way, Tong(王桐) can work for this. Tong is responsible for the
> automatically daily JStorm test, the only problem is that the
> automatically daily JStorm test is basing on Alibaba testing framework.
> Some of the components can't run on outside.
> But we can share the test cases firstly. Then we will try to port the old
> test cases to the storm tests.
>
> regardsLongda
> ------------------------------------------------------------------From:P.
> Taylor Goetz <pt...@gmail.com>Send Time:2015年11月6日(星期五)
> 12:12To:dev@storm.apache.org <de...@storm.apache.org>Subject:Re:
> [Discusson] Storm System Tests
> +1
> I think we should experiment with this. If it can serve us well, and we
> all agree, then we should consider adopting it.
>
> From what I see:
>
> * Icense is compatible
> * fairly well documented.
>
> What potentially worries me:
>
> * Maintenance. Can we count on updates? Is the ducktape community open to
> accepting patches from us? If so, what would be the turnaround time for
> acceptance?
>
> -Taylor
>
> > On Nov 5, 2015, at 8:54 PM, Harsha <st...@harsha.io> wrote:
> >
> > The reason for suggesting ducktape not just used in apache kafka but
> > also its getting security services integration like kdc and already has
> > zookeeper. This framework can work with vagrant vms or amazon ec2 etc..
> >
> >> On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
> >> Great, will make this a priority. Created a
> >> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
> >> assigned it to me.
> >>
> >> https://issues.apache.org/jira/browse/STORM-1179
> >>
> >>
> >> On Nov 5, 2015, at 11:40 AM, Bobby Evans
> >> <ev...@yahoo-inc.com>> wrote:
> >>
> >> Hugo,
> >>
> >> I would love to see that happen. It has been on my list for a while, but
> >> I have never found the time to do it. I personally am +1 on this,
> >> hopefully we can do this quickly in preparation for a 0.11.0 release.
> >>
> >> - Bobby
> >>
> >>
> >>
> >> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
> >> <hl...@hortonworks.com>> wrote:
> >>
> >>
> >> I a agree with these three levels of testing, and that at the very least
> >> we should keep unit tests separated from the system/integration tests.
> >> One huge advantage would be to quickly run unit tests that hopefully are
> >> more predictable, and thus avoid intermittent test fails.
> >>
> >> Bobby just mentioned but I had already in mind that it would be useful to
> >> create different maven profiles for integration and unit tests. I have
> >> done something similar in a different project and it works really well.
> >> If we agree that it is something we want to implement here, I will create
> >> a JIRA ticket for this and get it done. Please let me know.
> >>
> >> Thanks,
> >> Hugo
> >>
> >>> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
> >>>
> >>> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
> >>> I personally would like to see three levels of testing.
> >>> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
> >>> These would be run frequently but not necessarily a part of CI initially.
> >>> - Bobby
> >>>
> >>>
> >>> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io>> wrote:
> >>>
> >>>
> >>> Hi All,
> >>> As community we are growing and adding new and exciting
> >>> features to Storm and also we've ever growing connector which
> >>> only helps in storm adoption. One thing we've severely lacking
> >>> is system tests. There are unit tests which acts as
> >>> integration tests to storm-core but there are no system wide
> >>> integration tests that can spin up kafka nodes and storm ,
> >>> hbase and run a topology that can make sure the data is
> >>> getting into hbase.
> >>> We at Hortonworks use our test topologies run some of these
> >>> tests but this integration code to spin up vms or nodes is hard
> >>> to share. In apache kafka we are using ducktape to write system
> >>> tests so far its working out good. If there are any other
> >>> frameworks you've in mind we can definitely take a look. But as
> >>> a community we need start looking at
> >>> https://github.com/confluentinc/ducktape or similar frameworks
> >>> to start writing systems tests. Ducktape makes it easy to run
> >>> in a vm or any other infrastructure. Appreciate any feedback
> >>> on this.
> >>>
> >>> Thanks,
> >>> Harsha
> >>
> >>
> >>
Re: [Discusson] Storm System Tests
Posted by "封仲淹(纪君祥LongdaFeng)" <zh...@alibaba-inc.com>.
+1
By the way, Tong(王桐) can work for this. Tong is responsible for the automatically daily JStorm test, the only problem is that the automatically daily JStorm test is basing on Alibaba testing framework. Some of the components can't run on outside.
But we can share the test cases firstly. Then we will try to port the old test cases to the storm tests.
regardsLongda
------------------------------------------------------------------From:P. Taylor Goetz <pt...@gmail.com>Send Time:2015年11月6日(星期五) 12:12To:dev@storm.apache.org <de...@storm.apache.org>Subject:Re: [Discusson] Storm System Tests
+1
I think we should experiment with this. If it can serve us well, and we all agree, then we should consider adopting it.
From what I see:
* Icense is compatible
* fairly well documented.
What potentially worries me:
* Maintenance. Can we count on updates? Is the ducktape community open to accepting patches from us? If so, what would be the turnaround time for acceptance?
-Taylor
> On Nov 5, 2015, at 8:54 PM, Harsha <st...@harsha.io> wrote:
>
> The reason for suggesting ducktape not just used in apache kafka but
> also its getting security services integration like kdc and already has
> zookeeper. This framework can work with vagrant vms or amazon ec2 etc..
>
>> On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
>> Great, will make this a priority. Created a
>> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
>> assigned it to me.
>>
>> https://issues.apache.org/jira/browse/STORM-1179
>>
>>
>> On Nov 5, 2015, at 11:40 AM, Bobby Evans
>> <ev...@yahoo-inc.com>> wrote:
>>
>> Hugo,
>>
>> I would love to see that happen. It has been on my list for a while, but
>> I have never found the time to do it. I personally am +1 on this,
>> hopefully we can do this quickly in preparation for a 0.11.0 release.
>>
>> - Bobby
>>
>>
>>
>> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
>> <hl...@hortonworks.com>> wrote:
>>
>>
>> I a agree with these three levels of testing, and that at the very least
>> we should keep unit tests separated from the system/integration tests.
>> One huge advantage would be to quickly run unit tests that hopefully are
>> more predictable, and thus avoid intermittent test fails.
>>
>> Bobby just mentioned but I had already in mind that it would be useful to
>> create different maven profiles for integration and unit tests. I have
>> done something similar in a different project and it works really well.
>> If we agree that it is something we want to implement here, I will create
>> a JIRA ticket for this and get it done. Please let me know.
>>
>> Thanks,
>> Hugo
>>
>>> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
>>>
>>> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
>>> I personally would like to see three levels of testing.
>>> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
>>> These would be run frequently but not necessarily a part of CI initially.
>>> - Bobby
>>>
>>>
>>> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io>> wrote:
>>>
>>>
>>> Hi All,
>>> As community we are growing and adding new and exciting
>>> features to Storm and also we've ever growing connector which
>>> only helps in storm adoption. One thing we've severely lacking
>>> is system tests. There are unit tests which acts as
>>> integration tests to storm-core but there are no system wide
>>> integration tests that can spin up kafka nodes and storm ,
>>> hbase and run a topology that can make sure the data is
>>> getting into hbase.
>>> We at Hortonworks use our test topologies run some of these
>>> tests but this integration code to spin up vms or nodes is hard
>>> to share. In apache kafka we are using ducktape to write system
>>> tests so far its working out good. If there are any other
>>> frameworks you've in mind we can definitely take a look. But as
>>> a community we need start looking at
>>> https://github.com/confluentinc/ducktape or similar frameworks
>>> to start writing systems tests. Ducktape makes it easy to run
>>> in a vm or any other infrastructure. Appreciate any feedback
>>> on this.
>>>
>>> Thanks,
>>> Harsha
>>
>>
>>
Re: [Discusson] Storm System Tests
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
+1
I think we should experiment with this. If it can serve us well, and we all agree, then we should consider adopting it.
From what I see:
* Icense is compatible
* fairly well documented.
What potentially worries me:
* Maintenance. Can we count on updates? Is the ducktape community open to accepting patches from us? If so, what would be the turnaround time for acceptance?
-Taylor
> On Nov 5, 2015, at 8:54 PM, Harsha <st...@harsha.io> wrote:
>
> The reason for suggesting ducktape not just used in apache kafka but
> also its getting security services integration like kdc and already has
> zookeeper. This framework can work with vagrant vms or amazon ec2 etc..
>
>> On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
>> Great, will make this a priority. Created a
>> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
>> assigned it to me.
>>
>> https://issues.apache.org/jira/browse/STORM-1179
>>
>>
>> On Nov 5, 2015, at 11:40 AM, Bobby Evans
>> <ev...@yahoo-inc.com>> wrote:
>>
>> Hugo,
>>
>> I would love to see that happen. It has been on my list for a while, but
>> I have never found the time to do it. I personally am +1 on this,
>> hopefully we can do this quickly in preparation for a 0.11.0 release.
>>
>> - Bobby
>>
>>
>>
>> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
>> <hl...@hortonworks.com>> wrote:
>>
>>
>> I a agree with these three levels of testing, and that at the very least
>> we should keep unit tests separated from the system/integration tests.
>> One huge advantage would be to quickly run unit tests that hopefully are
>> more predictable, and thus avoid intermittent test fails.
>>
>> Bobby just mentioned but I had already in mind that it would be useful to
>> create different maven profiles for integration and unit tests. I have
>> done something similar in a different project and it works really well.
>> If we agree that it is something we want to implement here, I will create
>> a JIRA ticket for this and get it done. Please let me know.
>>
>> Thanks,
>> Hugo
>>
>>> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
>>>
>>> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
>>> I personally would like to see three levels of testing.
>>> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
>>> These would be run frequently but not necessarily a part of CI initially.
>>> - Bobby
>>>
>>>
>>> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io>> wrote:
>>>
>>>
>>> Hi All,
>>> As community we are growing and adding new and exciting
>>> features to Storm and also we've ever growing connector which
>>> only helps in storm adoption. One thing we've severely lacking
>>> is system tests. There are unit tests which acts as
>>> integration tests to storm-core but there are no system wide
>>> integration tests that can spin up kafka nodes and storm ,
>>> hbase and run a topology that can make sure the data is
>>> getting into hbase.
>>> We at Hortonworks use our test topologies run some of these
>>> tests but this integration code to spin up vms or nodes is hard
>>> to share. In apache kafka we are using ducktape to write system
>>> tests so far its working out good. If there are any other
>>> frameworks you've in mind we can definitely take a look. But as
>>> a community we need start looking at
>>> https://github.com/confluentinc/ducktape or similar frameworks
>>> to start writing systems tests. Ducktape makes it easy to run
>>> in a vm or any other infrastructure. Appreciate any feedback
>>> on this.
>>>
>>> Thanks,
>>> Harsha
>>
>>
>>
Re: [Discusson] Storm System Tests
Posted by Harsha <st...@harsha.io>.
The reason for suggesting ducktape not just used in apache kafka but
also its getting security services integration like kdc and already has
zookeeper. This framework can work with vagrant vms or amazon ec2 etc..
On Thu, Nov 5, 2015, at 11:49 AM, Hugo Da Cruz Louro wrote:
> Great, will make this a priority. Created a
> JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and
> assigned it to me.
>
> https://issues.apache.org/jira/browse/STORM-1179
>
>
> On Nov 5, 2015, at 11:40 AM, Bobby Evans
> <ev...@yahoo-inc.com>> wrote:
>
> Hugo,
>
> I would love to see that happen. It has been on my list for a while, but
> I have never found the time to do it. I personally am +1 on this,
> hopefully we can do this quickly in preparation for a 0.11.0 release.
>
> - Bobby
>
>
>
> On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro
> <hl...@hortonworks.com>> wrote:
>
>
> I a agree with these three levels of testing, and that at the very least
> we should keep unit tests separated from the system/integration tests.
> One huge advantage would be to quickly run unit tests that hopefully are
> more predictable, and thus avoid intermittent test fails.
>
> Bobby just mentioned but I had already in mind that it would be useful to
> create different maven profiles for integration and unit tests. I have
> done something similar in a different project and it works really well.
> If we agree that it is something we want to implement here, I will create
> a JIRA ticket for this and get it done. Please let me know.
>
> Thanks,
> Hugo
>
> > On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
> >
> > I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
> > I personally would like to see three levels of testing.
> > 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
> > These would be run frequently but not necessarily a part of CI initially.
> > - Bobby
> >
> >
> > On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io>> wrote:
> >
> >
> > Hi All,
> > As community we are growing and adding new and exciting
> > features to Storm and also we've ever growing connector which
> > only helps in storm adoption. One thing we've severely lacking
> > is system tests. There are unit tests which acts as
> > integration tests to storm-core but there are no system wide
> > integration tests that can spin up kafka nodes and storm ,
> > hbase and run a topology that can make sure the data is
> > getting into hbase.
> > We at Hortonworks use our test topologies run some of these
> > tests but this integration code to spin up vms or nodes is hard
> > to share. In apache kafka we are using ducktape to write system
> > tests so far its working out good. If there are any other
> > frameworks you've in mind we can definitely take a look. But as
> > a community we need start looking at
> > https://github.com/confluentinc/ducktape or similar frameworks
> > to start writing systems tests. Ducktape makes it easy to run
> > in a vm or any other infrastructure. Appreciate any feedback
> > on this.
> >
> > Thanks,
> > Harsha
> >
> >
>
>
>
Re: [Discusson] Storm System Tests
Posted by Hugo Da Cruz Louro <hl...@hortonworks.com>.
Great, will make this a priority. Created a JIRA<https://issues.apache.org/jira/browse/STORM-1179> ticket and assigned it to me.
https://issues.apache.org/jira/browse/STORM-1179
On Nov 5, 2015, at 11:40 AM, Bobby Evans <ev...@yahoo-inc.com>> wrote:
Hugo,
I would love to see that happen. It has been on my list for a while, but I have never found the time to do it. I personally am +1 on this, hopefully we can do this quickly in preparation for a 0.11.0 release.
- Bobby
On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro <hl...@hortonworks.com>> wrote:
I a agree with these three levels of testing, and that at the very least we should keep unit tests separated from the system/integration tests. One huge advantage would be to quickly run unit tests that hopefully are more predictable, and thus avoid intermittent test fails.
Bobby just mentioned but I had already in mind that it would be useful to create different maven profiles for integration and unit tests. I have done something similar in a different project and it works really well. If we agree that it is something we want to implement here, I will create a JIRA ticket for this and get it done. Please let me know.
Thanks,
Hugo
> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID>> wrote:
>
> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
> I personally would like to see three levels of testing.
> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
> These would be run frequently but not necessarily a part of CI initially.
> - Bobby
>
>
> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io>> wrote:
>
>
> Hi All,
> As community we are growing and adding new and exciting
> features to Storm and also we've ever growing connector which
> only helps in storm adoption. One thing we've severely lacking
> is system tests. There are unit tests which acts as
> integration tests to storm-core but there are no system wide
> integration tests that can spin up kafka nodes and storm ,
> hbase and run a topology that can make sure the data is
> getting into hbase.
> We at Hortonworks use our test topologies run some of these
> tests but this integration code to spin up vms or nodes is hard
> to share. In apache kafka we are using ducktape to write system
> tests so far its working out good. If there are any other
> frameworks you've in mind we can definitely take a look. But as
> a community we need start looking at
> https://github.com/confluentinc/ducktape or similar frameworks
> to start writing systems tests. Ducktape makes it easy to run
> in a vm or any other infrastructure. Appreciate any feedback
> on this.
>
> Thanks,
> Harsha
>
>
Re: [Discusson] Storm System Tests
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Hugo,
I would love to see that happen. It has been on my list for a while, but I have never found the time to do it. I personally am +1 on this, hopefully we can do this quickly in preparation for a 0.11.0 release.
- Bobby
On Thursday, November 5, 2015 1:37 PM, Hugo Da Cruz Louro <hl...@hortonworks.com> wrote:
I a agree with these three levels of testing, and that at the very least we should keep unit tests separated from the system/integration tests. One huge advantage would be to quickly run unit tests that hopefully are more predictable, and thus avoid intermittent test fails.
Bobby just mentioned but I had already in mind that it would be useful to create different maven profiles for integration and unit tests. I have done something similar in a different project and it works really well. If we agree that it is something we want to implement here, I will create a JIRA ticket for this and get it done. Please let me know.
Thanks,
Hugo
> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
> I personally would like to see three levels of testing.
> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
> These would be run frequently but not necessarily a part of CI initially.
> - Bobby
>
>
> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io> wrote:
>
>
> Hi All,
> As community we are growing and adding new and exciting
> features to Storm and also we've ever growing connector which
> only helps in storm adoption. One thing we've severely lacking
> is system tests. There are unit tests which acts as
> integration tests to storm-core but there are no system wide
> integration tests that can spin up kafka nodes and storm ,
> hbase and run a topology that can make sure the data is
> getting into hbase.
> We at Hortonworks use our test topologies run some of these
> tests but this integration code to spin up vms or nodes is hard
> to share. In apache kafka we are using ducktape to write system
> tests so far its working out good. If there are any other
> frameworks you've in mind we can definitely take a look. But as
> a community we need start looking at
> https://github.com/confluentinc/ducktape or similar frameworks
> to start writing systems tests. Ducktape makes it easy to run
> in a vm or any other infrastructure. Appreciate any feedback
> on this.
>
> Thanks,
> Harsha
>
>
Re: [Discusson] Storm System Tests
Posted by Hugo Da Cruz Louro <hl...@hortonworks.com>.
I a agree with these three levels of testing, and that at the very least we should keep unit tests separated from the system/integration tests. One huge advantage would be to quickly run unit tests that hopefully are more predictable, and thus avoid intermittent test fails.
Bobby just mentioned but I had already in mind that it would be useful to create different maven profiles for integration and unit tests. I have done something similar in a different project and it works really well. If we agree that it is something we want to implement here, I will create a JIRA ticket for this and get it done. Please let me know.
Thanks,
Hugo
> On Nov 5, 2015, at 11:23 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
>
> I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
> I personally would like to see three levels of testing.
> 1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
> These would be run frequently but not necessarily a part of CI initially.
> - Bobby
>
>
> On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io> wrote:
>
>
> Hi All,
> As community we are growing and adding new and exciting
> features to Storm and also we've ever growing connector which
> only helps in storm adoption. One thing we've severely lacking
> is system tests. There are unit tests which acts as
> integration tests to storm-core but there are no system wide
> integration tests that can spin up kafka nodes and storm ,
> hbase and run a topology that can make sure the data is
> getting into hbase.
> We at Hortonworks use our test topologies run some of these
> tests but this integration code to spin up vms or nodes is hard
> to share. In apache kafka we are using ducktape to write system
> tests so far its working out good. If there are any other
> frameworks you've in mind we can definitely take a look. But as
> a community we need start looking at
> https://github.com/confluentinc/ducktape or similar frameworks
> to start writing systems tests. Ducktape makes it easy to run
> in a vm or any other infrastructure. Appreciate any feedback
> on this.
>
> Thanks,
> Harsha
>
>
Re: [Discusson] Storm System Tests
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
+1 for different targets.
Thanks
Parth
On 11/5/15, 11:40 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:
>Our internal tests do things like that but as Harsha mentioned It is kind
>of tightly couple with our infra.
>
>I think if we can agree on using DuckTape (or some other framework) we
>can move to writing some simple test cases.
>
>Thanks
>Parth
>
>From: Bobby Evans <ev...@yahoo-inc.com>>
>Reply-To: Bobby Evans <ev...@yahoo-inc.com>>
>Date: Thursday, November 5, 2015 at 12:34 PM
>To: Parth Brahmbhatt
><pb...@hortonworks.com>>,
>"dev@storm.apache.org<ma...@storm.apache.org>"
><de...@storm.apache.org>>
>Subject: Re: [Discusson] Storm System Tests
>
>If you have a patch that just brings up a very small cluster with 2
>nimbus instances and shoots things, that would be really great.
>
>- Bobby
>
>
>
>On Thursday, November 5, 2015 1:29 PM, Parth Brahmbhatt
><pb...@hortonworks.com>> wrote:
>
>
>It will be good to add a basic test suite before 0.11 release at least to
>test out things like HA where we should really have some chaos monkey
>testing. From all the options I have seen/used duck tape seems to be the
>best option as of now.
>
>Thanks
>Parth
>
>On 11/5/15, 11:23 AM, "Bobby Evans"
><ev...@yahoo-inc.com.INVALID>> wrote:
>
>>These would be run frequently but not necessarily a part of CI initially.
>>
>
>
Re: [Discusson] Storm System Tests
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
Our internal tests do things like that but as Harsha mentioned It is kind of tightly couple with our infra.
I think if we can agree on using DuckTape (or some other framework) we can move to writing some simple test cases.
Thanks
Parth
From: Bobby Evans <ev...@yahoo-inc.com>>
Reply-To: Bobby Evans <ev...@yahoo-inc.com>>
Date: Thursday, November 5, 2015 at 12:34 PM
To: Parth Brahmbhatt <pb...@hortonworks.com>>, "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Subject: Re: [Discusson] Storm System Tests
If you have a patch that just brings up a very small cluster with 2 nimbus instances and shoots things, that would be really great.
- Bobby
On Thursday, November 5, 2015 1:29 PM, Parth Brahmbhatt <pb...@hortonworks.com>> wrote:
It will be good to add a basic test suite before 0.11 release at least to
test out things like HA where we should really have some chaos monkey
testing. From all the options I have seen/used duck tape seems to be the
best option as of now.
Thanks
Parth
On 11/5/15, 11:23 AM, "Bobby Evans" <ev...@yahoo-inc.com.INVALID>> wrote:
>These would be run frequently but not necessarily a part of CI initially.
>
Re: [Discusson] Storm System Tests
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
If you have a patch that just brings up a very small cluster with 2 nimbus instances and shoots things, that would be really great.
- Bobby
On Thursday, November 5, 2015 1:29 PM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
It will be good to add a basic test suite before 0.11 release at least to
test out things like HA where we should really have some chaos monkey
testing. From all the options I have seen/used duck tape seems to be the
best option as of now.
Thanks
Parth
On 11/5/15, 11:23 AM, "Bobby Evans" <ev...@yahoo-inc.com.INVALID> wrote:
>These would be run frequently but not necessarily a part of CI initially.
>
Re: [Discusson] Storm System Tests
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
It will be good to add a basic test suite before 0.11 release at least to
test out things like HA where we should really have some chaos monkey
testing. From all the options I have seen/used duck tape seems to be the
best option as of now.
Thanks
Parth
On 11/5/15, 11:23 AM, "Bobby Evans" <ev...@yahoo-inc.com.INVALID> wrote:
>These would be run frequently but not necessarily a part of CI initially.
>
Re: [Discusson] Storm System Tests
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
I totally agree. Too many of our "unit" tests are integration tests and end up spinning up an entire local-mode cluster.
I personally would like to see three levels of testing.
1) true unit tests. They only touch the code under test and very little else. The should be what you get when you run mvn test2) integration tests. These should spin up local mode clusters and modify configs/etc to get a decent set of more white box tests. The should run as a part of trivis-ci, and probably should run by enabling a special profile.3) Sanity Integration Tests. ducktape looks like a great fit here. I would love to see us spin up an few different scenarios for testing, with/without security. Talking to Kafka, Hadoop(HBase, HDFS, Hive), redis, elastasearch, etc.
These would be run frequently but not necessarily a part of CI initially.
- Bobby
On Wednesday, November 4, 2015 7:21 PM, Harsha <st...@harsha.io> wrote:
Hi All,
As community we are growing and adding new and exciting
features to Storm and also we've ever growing connector which
only helps in storm adoption. One thing we've severely lacking
is system tests. There are unit tests which acts as
integration tests to storm-core but there are no system wide
integration tests that can spin up kafka nodes and storm ,
hbase and run a topology that can make sure the data is
getting into hbase.
We at Hortonworks use our test topologies run some of these
tests but this integration code to spin up vms or nodes is hard
to share. In apache kafka we are using ducktape to write system
tests so far its working out good. If there are any other
frameworks you've in mind we can definitely take a look. But as
a community we need start looking at
https://github.com/confluentinc/ducktape or similar frameworks
to start writing systems tests. Ducktape makes it easy to run
in a vm or any other infrastructure. Appreciate any feedback
on this.
Thanks,
Harsha