You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Wilder Rodrigues <WR...@schubergphilis.com> on 2015/09/15 14:18:59 UTC
VPC router issue
Hi all,
This morning I shared a message on our Slack channel about the current issue I’m facing with VPC routers. The problem was caught by the test_vpc_routers.py tests, which are unable to verify the following:
* Create a VPC with 2 Tiers, 3 VMs, 2Pub IPs, and 1 ACL
* Stop the VPC router
* Start the VPC router
I have been busy trying to figure out when it stopped working when I then stopped at commit https://github.com/apache/cloudstack/commit/f5e5f4d0026f8ffd6f3aa7e8e4c7be0cd809d6c9 - although it’s not really the one causing the issue given what I have changed there.
The only way to get it working is by restarting the VPC with the cleanup option - os destroying the VPC router and just restarting the VPC.
Since not many people have been testing the routers as much as I do, it seems no one is aware of this issue.
Long story short: current master is broken.
/me digging through PRs to find the root cause.
Cheers,
Wilder
Re: VPC router issue
Posted by Wilder Rodrigues <WR...@schubergphilis.com>.
Thanks for the effort, Raja!
I’m testing other PRs trying to find where the problem was introduced. Will keep you posted.
Cheers,
Wilder
On 16 Sep 2015, at 06:35, Raja Pullela <ra...@citrix.com>> wrote:
Hi Wilder, BTW, I just checked my setup and it was passing on Sep 9th and was regressed after that.
Thanks for checking this out,
Raja
-----Original Message-----
From: Wilder Rodrigues [mailto:WRodrigues@schubergphilis.com]
Sent: Tuesday, September 15, 2015 7:12 PM
To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
Cc: Remi Bergsma <RB...@schubergphilis.com>>; Rohit Yadav <ro...@shapeblue.com>>; Rajani Karuturi <ra...@apache.org>>; Daan Hoogland <da...@gmail.com>>
Subject: Re: VPC router issue
6 days ago test_pc_routers.py succeeded with PR 788:
https://github.com/apache/cloudstack/pull/788
It’s in the test results I put there.
I’m now testing Master to verify if it’s still occurring there. I tested the PR #808 (https://github.com/apache/cloudstack/pull/805), which had a bunch of tests executed against, but not the test_vpc_routers, and there it fails.
So, the problem occurred between 788 and 805.
Will keep digging.
Cheers,
Wilder
On 15 Sep 2015, at 14:18, Wilder Rodrigues <WR...@schubergphilis.com>> wrote:
Hi all,
This morning I shared a message on our Slack channel about the current issue I’m facing with VPC routers. The problem was caught by the test_vpc_routers.py tests, which are unable to verify the following:
* Create a VPC with 2 Tiers, 3 VMs, 2Pub IPs, and 1 ACL
* Stop the VPC router
* Start the VPC router
I have been busy trying to figure out when it stopped working when I then stopped at commit https://github.com/apache/cloudstack/commit/f5e5f4d0026f8ffd6f3aa7e8e4c7be0cd809d6c9 - although it’s not really the one causing the issue given what I have changed there.
The only way to get it working is by restarting the VPC with the cleanup option - os destroying the VPC router and just restarting the VPC.
Since not many people have been testing the routers as much as I do, it seems no one is aware of this issue.
Long story short: current master is broken.
/me digging through PRs to find the root cause.
Cheers,
Wilder
Re: VPC router issue
Posted by Wilder Rodrigues <WR...@schubergphilis.com>.
Issue is here: https://issues.apache.org/jira/browse/CLOUDSTACK-8878
Cheers,
Wilder
On 17 Sep 2015, at 10:14, Wilder Rodrigues <WR...@schubergphilis.com>> wrote:
Hi Rajani,
The last commit from 8 September works: https://github.com/apache/cloudstack/commit/5e9e9b84fb1a4ca029c32e5b1c305124bfa4d4af
The merge commit just before the 5e9e9b does NOT work: https://github.com/apache/cloudstack/commit/237e0e4611dd3d0ae56a8c0557dba93e7fbd3440
And the commit before it WORKS: https://github.com/apache/cloudstack/commit/30792532ed0f3f0622ac3ef6856d9b7d7a48d1d0. That’s the commit of PR #777, which is the parent of PR #765
So I believe we have a good candidate! The fix for CLOUDSTACK-8688 on PR #765.
Apologies for the problems caused by the fix, although nobody complained except myself.
I will proceed with creating a blocker issue and fixing the problem.
Chers,
Wilder
On 16 Sep 2015, at 10:47, Rajani Karuturi <Ra...@citrix.com>> wrote:
Raja,
Do you know the commit ids between the last successful and first failed builds?
~Rajani
On 16-Sep-2015, at 10:05 am, Raja Pullela <ra...@citrix.com>> wrote:
Hi Wilder, BTW, I just checked my setup and it was passing on Sep 9th and was regressed after that.
Thanks for checking this out,
Raja
-----Original Message-----
From: Wilder Rodrigues [mailto:WRodrigues@schubergphilis.com]
Sent: Tuesday, September 15, 2015 7:12 PM
To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
Cc: Remi Bergsma <RB...@schubergphilis.com>>; Rohit Yadav <ro...@shapeblue.com>>; Rajani Karuturi <ra...@apache.org>>; Daan Hoogland <da...@gmail.com>>
Subject: Re: VPC router issue
6 days ago test_pc_routers.py succeeded with PR 788:
https://github.com/apache/cloudstack/pull/788
It’s in the test results I put there.
I’m now testing Master to verify if it’s still occurring there. I tested the PR #808 (https://github.com/apache/cloudstack/pull/805), which had a bunch of tests executed against, but not the test_vpc_routers, and there it fails.
So, the problem occurred between 788 and 805.
Will keep digging.
Cheers,
Wilder
On 15 Sep 2015, at 14:18, Wilder Rodrigues <WR...@schubergphilis.com>> wrote:
Hi all,
This morning I shared a message on our Slack channel about the current issue I’m facing with VPC routers. The problem was caught by the test_vpc_routers.py tests, which are unable to verify the following:
* Create a VPC with 2 Tiers, 3 VMs, 2Pub IPs, and 1 ACL
* Stop the VPC router
* Start the VPC router
I have been busy trying to figure out when it stopped working when I then stopped at commit https://github.com/apache/cloudstack/commit/f5e5f4d0026f8ffd6f3aa7e8e4c7be0cd809d6c9 - although it’s not really the one causing the issue given what I have changed there.
The only way to get it working is by restarting the VPC with the cleanup option - os destroying the VPC router and just restarting the VPC.
Since not many people have been testing the routers as much as I do, it seems no one is aware of this issue.
Long story short: current master is broken.
/me digging through PRs to find the root cause.
Cheers,
Wilder
Re: VPC router issue
Posted by Wilder Rodrigues <WR...@schubergphilis.com>.
Hi Rajani,
The last commit from 8 September works: https://github.com/apache/cloudstack/commit/5e9e9b84fb1a4ca029c32e5b1c305124bfa4d4af
The merge commit just before the 5e9e9b does NOT work: https://github.com/apache/cloudstack/commit/237e0e4611dd3d0ae56a8c0557dba93e7fbd3440
And the commit before it WORKS: https://github.com/apache/cloudstack/commit/30792532ed0f3f0622ac3ef6856d9b7d7a48d1d0. That’s the commit of PR #777, which is the parent of PR #765
So I believe we have a good candidate! The fix for CLOUDSTACK-8688 on PR #765.
Apologies for the problems caused by the fix, although nobody complained except myself.
I will proceed with creating a blocker issue and fixing the problem.
Chers,
Wilder
On 16 Sep 2015, at 10:47, Rajani Karuturi <Ra...@citrix.com>> wrote:
Raja,
Do you know the commit ids between the last successful and first failed builds?
~Rajani
On 16-Sep-2015, at 10:05 am, Raja Pullela <ra...@citrix.com>> wrote:
Hi Wilder, BTW, I just checked my setup and it was passing on Sep 9th and was regressed after that.
Thanks for checking this out,
Raja
-----Original Message-----
From: Wilder Rodrigues [mailto:WRodrigues@schubergphilis.com]
Sent: Tuesday, September 15, 2015 7:12 PM
To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
Cc: Remi Bergsma <RB...@schubergphilis.com>>; Rohit Yadav <ro...@shapeblue.com>>; Rajani Karuturi <ra...@apache.org>>; Daan Hoogland <da...@gmail.com>>
Subject: Re: VPC router issue
6 days ago test_pc_routers.py succeeded with PR 788:
https://github.com/apache/cloudstack/pull/788
It’s in the test results I put there.
I’m now testing Master to verify if it’s still occurring there. I tested the PR #808 (https://github.com/apache/cloudstack/pull/805), which had a bunch of tests executed against, but not the test_vpc_routers, and there it fails.
So, the problem occurred between 788 and 805.
Will keep digging.
Cheers,
Wilder
On 15 Sep 2015, at 14:18, Wilder Rodrigues <WR...@schubergphilis.com>> wrote:
Hi all,
This morning I shared a message on our Slack channel about the current issue I’m facing with VPC routers. The problem was caught by the test_vpc_routers.py tests, which are unable to verify the following:
* Create a VPC with 2 Tiers, 3 VMs, 2Pub IPs, and 1 ACL
* Stop the VPC router
* Start the VPC router
I have been busy trying to figure out when it stopped working when I then stopped at commit https://github.com/apache/cloudstack/commit/f5e5f4d0026f8ffd6f3aa7e8e4c7be0cd809d6c9 - although it’s not really the one causing the issue given what I have changed there.
The only way to get it working is by restarting the VPC with the cleanup option - os destroying the VPC router and just restarting the VPC.
Since not many people have been testing the routers as much as I do, it seems no one is aware of this issue.
Long story short: current master is broken.
/me digging through PRs to find the root cause.
Cheers,
Wilder
Re: VPC router issue
Posted by Rajani Karuturi <Ra...@citrix.com>.
Raja,
Do you know the commit ids between the last successful and first failed builds?
~Rajani
On 16-Sep-2015, at 10:05 am, Raja Pullela <ra...@citrix.com> wrote:
> Hi Wilder, BTW, I just checked my setup and it was passing on Sep 9th and was regressed after that.
>
> Thanks for checking this out,
> Raja
> -----Original Message-----
> From: Wilder Rodrigues [mailto:WRodrigues@schubergphilis.com]
> Sent: Tuesday, September 15, 2015 7:12 PM
> To: dev@cloudstack.apache.org
> Cc: Remi Bergsma <RB...@schubergphilis.com>; Rohit Yadav <ro...@shapeblue.com>; Rajani Karuturi <ra...@apache.org>; Daan Hoogland <da...@gmail.com>
> Subject: Re: VPC router issue
>
> 6 days ago test_pc_routers.py succeeded with PR 788:
>
> https://github.com/apache/cloudstack/pull/788
>
> It’s in the test results I put there.
>
> I’m now testing Master to verify if it’s still occurring there. I tested the PR #808 (https://github.com/apache/cloudstack/pull/805), which had a bunch of tests executed against, but not the test_vpc_routers, and there it fails.
>
> So, the problem occurred between 788 and 805.
>
> Will keep digging.
>
> Cheers,
> Wilder
>
>
> On 15 Sep 2015, at 14:18, Wilder Rodrigues <WR...@schubergphilis.com>> wrote:
>
> Hi all,
>
> This morning I shared a message on our Slack channel about the current issue I’m facing with VPC routers. The problem was caught by the test_vpc_routers.py tests, which are unable to verify the following:
>
> * Create a VPC with 2 Tiers, 3 VMs, 2Pub IPs, and 1 ACL
> * Stop the VPC router
> * Start the VPC router
>
> I have been busy trying to figure out when it stopped working when I then stopped at commit https://github.com/apache/cloudstack/commit/f5e5f4d0026f8ffd6f3aa7e8e4c7be0cd809d6c9 - although it’s not really the one causing the issue given what I have changed there.
>
> The only way to get it working is by restarting the VPC with the cleanup option - os destroying the VPC router and just restarting the VPC.
>
> Since not many people have been testing the routers as much as I do, it seems no one is aware of this issue.
>
> Long story short: current master is broken.
>
> /me digging through PRs to find the root cause.
>
> Cheers,
> Wilder
>
RE: VPC router issue
Posted by Raja Pullela <ra...@citrix.com>.
Hi Wilder, BTW, I just checked my setup and it was passing on Sep 9th and was regressed after that.
Thanks for checking this out,
Raja
-----Original Message-----
From: Wilder Rodrigues [mailto:WRodrigues@schubergphilis.com]
Sent: Tuesday, September 15, 2015 7:12 PM
To: dev@cloudstack.apache.org
Cc: Remi Bergsma <RB...@schubergphilis.com>; Rohit Yadav <ro...@shapeblue.com>; Rajani Karuturi <ra...@apache.org>; Daan Hoogland <da...@gmail.com>
Subject: Re: VPC router issue
6 days ago test_pc_routers.py succeeded with PR 788:
https://github.com/apache/cloudstack/pull/788
It’s in the test results I put there.
I’m now testing Master to verify if it’s still occurring there. I tested the PR #808 (https://github.com/apache/cloudstack/pull/805), which had a bunch of tests executed against, but not the test_vpc_routers, and there it fails.
So, the problem occurred between 788 and 805.
Will keep digging.
Cheers,
Wilder
On 15 Sep 2015, at 14:18, Wilder Rodrigues <WR...@schubergphilis.com>> wrote:
Hi all,
This morning I shared a message on our Slack channel about the current issue I’m facing with VPC routers. The problem was caught by the test_vpc_routers.py tests, which are unable to verify the following:
* Create a VPC with 2 Tiers, 3 VMs, 2Pub IPs, and 1 ACL
* Stop the VPC router
* Start the VPC router
I have been busy trying to figure out when it stopped working when I then stopped at commit https://github.com/apache/cloudstack/commit/f5e5f4d0026f8ffd6f3aa7e8e4c7be0cd809d6c9 - although it’s not really the one causing the issue given what I have changed there.
The only way to get it working is by restarting the VPC with the cleanup option - os destroying the VPC router and just restarting the VPC.
Since not many people have been testing the routers as much as I do, it seems no one is aware of this issue.
Long story short: current master is broken.
/me digging through PRs to find the root cause.
Cheers,
Wilder
Re: VPC router issue
Posted by Wilder Rodrigues <WR...@schubergphilis.com>.
6 days ago test_pc_routers.py succeeded with PR 788:
https://github.com/apache/cloudstack/pull/788
It’s in the test results I put there.
I’m now testing Master to verify if it’s still occurring there. I tested the PR #808 (https://github.com/apache/cloudstack/pull/805), which had a bunch of tests executed against, but not the test_vpc_routers, and there it fails.
So, the problem occurred between 788 and 805.
Will keep digging.
Cheers,
Wilder
On 15 Sep 2015, at 14:18, Wilder Rodrigues <WR...@schubergphilis.com>> wrote:
Hi all,
This morning I shared a message on our Slack channel about the current issue I’m facing with VPC routers. The problem was caught by the test_vpc_routers.py tests, which are unable to verify the following:
* Create a VPC with 2 Tiers, 3 VMs, 2Pub IPs, and 1 ACL
* Stop the VPC router
* Start the VPC router
I have been busy trying to figure out when it stopped working when I then stopped at commit https://github.com/apache/cloudstack/commit/f5e5f4d0026f8ffd6f3aa7e8e4c7be0cd809d6c9 - although it’s not really the one causing the issue given what I have changed there.
The only way to get it working is by restarting the VPC with the cleanup option - os destroying the VPC router and just restarting the VPC.
Since not many people have been testing the routers as much as I do, it seems no one is aware of this issue.
Long story short: current master is broken.
/me digging through PRs to find the root cause.
Cheers,
Wilder