You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Murali Reddy <mu...@gmail.com> on 2016/08/19 07:03:30 UTC
Virtual Router : Marvin test gap analysis
All,
We (at ShapeBlue) did a gap analysis to figure if current set of smoke and component tests sufficiently test the VR functionality for the regressions. I have posted the analysis at [1]. I went through the test suites, and listed down all the tests that touch virtual router functionality. There is listing of general observations on grey areas.
One particular area where there were no tests, was related multiple public IP’s from different public IP ranges associated with a network. From 4.6, all the way to master only IP’s from one public IP range (eth2 on the VR) is working, any network services on the public IP’s on eth3, eth4 etc on VR are not functional. This is a common use case and is broken for last few releases. Bug in this area are reported [2] and PR is yet to be merged [3]. I will be work on the patch to get this fixed in LTS. I have also added Marvin tests [4] cover multiple public IP scenarios.
Given the flexibility and rich set of network functionality is CloudStack, we could catch regression only if we have good test suite. If there are any other areas related to virtual router functionality that you see there are significant gaps, please chime in share your thoughts or add the the wiki.
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Virtual+Router%3A+Smoke+and+Integration+tests+gap+analysis
[2] https://issues.apache.org/jira/browse/CLOUDSTACK-9339
[3] https://github.com/apache/cloudstack/pull/1519
[4] https://github.com/murali-reddy/cloudstack/commit/0b6fbc29fcadb39b08d0050ca473680a614dfab4
Re: Virtual Router : Marvin test gap analysis
Posted by Murali Reddy <mu...@gmail.com>.
We (Abhi, Boris and me) opened couple of marvin test bugs to automate the gaps [1], we will start to work on them. If any one wishes to contribute please feel free to pick them up.
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Virtual+Router%3A+Smoke+and+Integration+tests+gap+analysis
On 26/08/16, 9:35 PM, "Murali Reddy" <mu...@gmail.com> wrote:
>Raja,
>
>Thanks for sharing insightful automation analysis. We are trying to figure out how to convert our efforts in to actionable items. As community we can work on them to get good automated test suite for VR.
>
>
>
>Will be sharing details early next week.
>
>Thanks.
>
>On 19/08/16, 2:25 PM, "Raja Pullela" <ra...@accelerite.com> wrote:
>
>>Hi Murali,
>>
>>Great initiative… and VR is one area that could help greatly if we have more automation. we have done high-level analysis on VR functionality/automation recently and have posted our findings to the wiki under “high-level Automation analysis” section. Please take a look,
>>
>>[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Virtual+Router%3A+Smoke+and+Integration+tests+gap+analysis
>>
>>Best,
>>Raja
>>Senior Manager, Product Development
>>Accelerite, www.accelerite.com, @accelerite
>>2055, Laurelwood Road, Santa Clara, CA 95054, USA
>>Phone: 1-408-216-7010
>>
>>On 8/19/16, 12:33 PM, "Murali Reddy" <mu...@gmail.com> wrote:
>>
>>All,
>>
>>We (at ShapeBlue) did a gap analysis to figure if current set of smoke and component tests sufficiently test the VR functionality for the regressions. I have posted the analysis at [1]. I went through the test suites, and listed down all the tests that touch virtual router functionality. There is listing of general observations on grey areas.
>>
>>One particular area where there were no tests, was related multiple public IP’s from different public IP ranges associated with a network. From 4.6, all the way to master only IP’s from one public IP range (eth2 on the VR) is working, any network services on the public IP’s on eth3, eth4 etc on VR are not functional. This is a common use case and is broken for last few releases. Bug in this area are reported [2] and PR is yet to be merged [3]. I will be work on the patch to get this fixed in LTS. I have also added Marvin tests [4] cover multiple public IP scenarios.
>>
>>Given the flexibility and rich set of network functionality is CloudStack, we could catch regression only if we have good test suite. If there are any other areas related to virtual router functionality that you see there are significant gaps, please chime in share your thoughts or add the the wiki.
>>
>>[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Virtual+Router%3A+Smoke+and+Integration+tests+gap+analysis
>>[2] https://issues.apache.org/jira/browse/CLOUDSTACK-9339
>>[3] https://github.com/apache/cloudstack/pull/1519
>>[4] https://github.com/murali-reddy/cloudstack/commit/0b6fbc29fcadb39b08d0050ca473680a614dfab4
>>
>>
>>
>>
>>
>>
>>
>>DISCLAIMER
>>==========
>>This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
>
Re: Virtual Router : Marvin test gap analysis
Posted by Murali Reddy <mu...@gmail.com>.
Raja,
Thanks for sharing insightful automation analysis. We are trying to figure out how to convert our efforts in to actionable items. As community we can work on them to get good automated test suite for VR.
Will be sharing details early next week.
Thanks.
On 19/08/16, 2:25 PM, "Raja Pullela" <ra...@accelerite.com> wrote:
>Hi Murali,
>
>Great initiative… and VR is one area that could help greatly if we have more automation. we have done high-level analysis on VR functionality/automation recently and have posted our findings to the wiki under “high-level Automation analysis” section. Please take a look,
>
>[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Virtual+Router%3A+Smoke+and+Integration+tests+gap+analysis
>
>Best,
>Raja
>Senior Manager, Product Development
>Accelerite, www.accelerite.com, @accelerite
>2055, Laurelwood Road, Santa Clara, CA 95054, USA
>Phone: 1-408-216-7010
>
>On 8/19/16, 12:33 PM, "Murali Reddy" <mu...@gmail.com> wrote:
>
>All,
>
>We (at ShapeBlue) did a gap analysis to figure if current set of smoke and component tests sufficiently test the VR functionality for the regressions. I have posted the analysis at [1]. I went through the test suites, and listed down all the tests that touch virtual router functionality. There is listing of general observations on grey areas.
>
>One particular area where there were no tests, was related multiple public IP’s from different public IP ranges associated with a network. From 4.6, all the way to master only IP’s from one public IP range (eth2 on the VR) is working, any network services on the public IP’s on eth3, eth4 etc on VR are not functional. This is a common use case and is broken for last few releases. Bug in this area are reported [2] and PR is yet to be merged [3]. I will be work on the patch to get this fixed in LTS. I have also added Marvin tests [4] cover multiple public IP scenarios.
>
>Given the flexibility and rich set of network functionality is CloudStack, we could catch regression only if we have good test suite. If there are any other areas related to virtual router functionality that you see there are significant gaps, please chime in share your thoughts or add the the wiki.
>
>[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Virtual+Router%3A+Smoke+and+Integration+tests+gap+analysis
>[2] https://issues.apache.org/jira/browse/CLOUDSTACK-9339
>[3] https://github.com/apache/cloudstack/pull/1519
>[4] https://github.com/murali-reddy/cloudstack/commit/0b6fbc29fcadb39b08d0050ca473680a614dfab4
>
>
>
>
>
>
>
>DISCLAIMER
>==========
>This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Virtual Router : Marvin test gap analysis
Posted by Raja Pullela <ra...@accelerite.com>.
Hi Murali,
Great initiative… and VR is one area that could help greatly if we have more automation. we have done high-level analysis on VR functionality/automation recently and have posted our findings to the wiki under “high-level Automation analysis” section. Please take a look,
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Virtual+Router%3A+Smoke+and+Integration+tests+gap+analysis
Best,
Raja
Senior Manager, Product Development
Accelerite, www.accelerite.com, @accelerite
2055, Laurelwood Road, Santa Clara, CA 95054, USA
Phone: 1-408-216-7010
On 8/19/16, 12:33 PM, "Murali Reddy" <mu...@gmail.com> wrote:
All,
We (at ShapeBlue) did a gap analysis to figure if current set of smoke and component tests sufficiently test the VR functionality for the regressions. I have posted the analysis at [1]. I went through the test suites, and listed down all the tests that touch virtual router functionality. There is listing of general observations on grey areas.
One particular area where there were no tests, was related multiple public IP’s from different public IP ranges associated with a network. From 4.6, all the way to master only IP’s from one public IP range (eth2 on the VR) is working, any network services on the public IP’s on eth3, eth4 etc on VR are not functional. This is a common use case and is broken for last few releases. Bug in this area are reported [2] and PR is yet to be merged [3]. I will be work on the patch to get this fixed in LTS. I have also added Marvin tests [4] cover multiple public IP scenarios.
Given the flexibility and rich set of network functionality is CloudStack, we could catch regression only if we have good test suite. If there are any other areas related to virtual router functionality that you see there are significant gaps, please chime in share your thoughts or add the the wiki.
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Virtual+Router%3A+Smoke+and+Integration+tests+gap+analysis
[2] https://issues.apache.org/jira/browse/CLOUDSTACK-9339
[3] https://github.com/apache/cloudstack/pull/1519
[4] https://github.com/murali-reddy/cloudstack/commit/0b6fbc29fcadb39b08d0050ca473680a614dfab4
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.