You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Girish Shilamkar <gs...@ddn.com> on 2013/10/16 07:11:46 UTC
Http connection failure in marvin
Hello,
Sometimes while running automation testcases tests just fail with following error:
ConnectionError: HTTPConnectionPool(host='10.147.59.65', port=8080): Max retries exceeded with url: /client/api? apiKey=lcgWQ9XrjyIyPjsl7Y0BGI2udSRMs6S1w_42lWkwpCSfLWjkgoWvCakWwUCJ3TqfgMmuHWT4DXv5Lgoz-KCuSQ&egressdefaultpolicy=true&serviceproviderlist%5B2%5D. provider=VirtualRouter&serviceproviderlist%5B5%5D.service=Dhcp&displaytext=Network+offering-VR+services-8FM4XI&specifyVlan=False&serviceproviderlist%5B4%5D. provider=VirtualRouter&serviceproviderlist%5B1%5D.provider=VirtualRouter&availability=Optional&conservemode=True&servicecapabilitylist%5B1%5D. capabilitytype=SupportedSourceNatTypes&serviceproviderlist%5B3%5D.service=Dns&serviceproviderlist%5B5%5D.provider=VirtualRouter&response=json&serviceproviderlist%5B0%5D. provider=VirtualRouter&servicecapabilitylist%5B0%5D.capabilitytype=RedundantRouter&serviceproviderlist%5B8%5D.provider=VirtualRouter&serviceproviderlist%5B1%5D. service=Lb&servicecapabilitylist%5B0%5D.service=SourceNat&serviceproviderlist%5B4%5D. service=Firewall&supportedservices=Dhcp%2CDns%2CSourceNat%2CPortForwarding%2CVpn%2CFirewall%2CLb%2CUserData%2CStaticNat&traffictype=GUEST&servicecapabilitylist%5B1%5D. service=SourceNat&serviceproviderlist%5B6%5D.provider=VirtualRouter&serviceproviderlist%5B8%5D.service=StaticNat&serviceproviderlist%5B3%5D.provider=VirtualRouter&name=Network+ offering-VR+services-OLND5Q&guestiptype=Isolated&serviceproviderlist%5B7%5D.provider=VirtualRouter&servicecapabilitylist%5B1%5D. capabilityvalue=peraccount&serviceproviderlist%5B7%5D.service=SourceNat&serviceproviderlist%5B2%5D.service=PortForwarding&servicecapabilitylist%5B0%5D. capabilityvalue=true&serviceproviderlist%5B6%5D.service=Vpn&command=createNetworkOffering&signature=dFBPTQoc%2B%2B42%2FhDVyQQOFB87c7k%3D&serviceproviderlist%5B0%5D.service=UserData (Caused by <class 'httplib.BadStatusLine'>: '')
These errors are intermittent. While investigating this problem I found this in marvin.cloudstackTestClient
def submitCmdsAndWait(self, cmds, workers=1):
'''FixME, httplib has issue if more than one thread submitted'''
if self.asyncJobMgr is None:
self.asyncJobMgr = asyncJobMgr.asyncJobMgr(self.apiClient,
self.dbConnection)
return self.asyncJobMgr.submitCmdsAndWait(cmds, workers)
My theory is if the response from management server slow, multiple thread are submitted and we hit this issue.
Please advise.
Regards,
Girish
Re: Http connection failure in marvin
Posted by Girish Shilamkar <gi...@clogeny.com>.
Thanks Santhosh for the pointers. I will investigate on these lines.
Regards,
Girish
On 16-Oct-2013, at 3:02 PM, Santhosh Edukulla <sa...@citrix.com> wrote:
> Girish,
>
> Thanks for the bug. Few notes below.
>
> 1/ Can you please add the server GET\POST calls with params received at the server information to the bug? This will help to know for which GET/POST call this issue has appeared?
>
>
> 2/ Under tools/marvin/marvin/cloudstackConnection.py, there is a member function "request" under cloudConnection class. If this exception is appearing quiet a good number of times. Can you please add this specific exception( BadStatusLine ) to be caught and collect few specific information for request sent,response recieved when that exception is caught in your next run? The total url with args and body to be dumped etc in a file?
>
> 3/ We are using requests library to make http calls under marvin, it seems there is a way to enable debug information for this library, check the below link. Please enable the debug information for this library and run the tests again to see by catching the complete request payload when this specific exception is raised.
>
> http://docs.python-requests.org/en/master/api/#api-changes
>
> # these two lines enable debugging at httplib level (requests->urllib3->httplib)
> # you will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
> # the only thing missing will be the response.body which is not logged.
> import httplib
> httplib.HTTPConnection.debuglevel = 1
>
>
> 4/ One of the link suggests a trailing new line character for the request as the cause for this issue. Adding complete server calls and debug information will help to debug more i believe?
>
> http://stackoverflow.com/questions/1767934/why-am-i-getting-this-error-in-python-httplib
>
> Check above and you can get more information for this issue to delve further.
>
>
> Thanks!
> Santhosh
> ________________________________________
> From: Girish Shilamkar [girish@clogeny.com]
> Sent: Wednesday, October 16, 2013 3:01 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Http connection failure in marvin
>
> Lets track this issue here https://issues.apache.org/jira/browse/CLOUDSTACK-4846
>
> Regards,
> Girish
>
> On 16-Oct-2013, at 11:30 AM, Prasanna Santhanam <ts...@apache.org> wrote:
>
>> httplib is used indirectly through requests. I'd try reproduce this
>> by throttling 1000s of requests through a single mgmt server. It is
>> intermittent because the mgmt server is probably loaded when you hit
>> it and possibly producing an invalid/empty response.
>
RE: Http connection failure in marvin
Posted by Santhosh Edukulla <sa...@citrix.com>.
Girish,
Thanks for the bug. Few notes below.
1/ Can you please add the server GET\POST calls with params received at the server information to the bug? This will help to know for which GET/POST call this issue has appeared?
2/ Under tools/marvin/marvin/cloudstackConnection.py, there is a member function "request" under cloudConnection class. If this exception is appearing quiet a good number of times. Can you please add this specific exception( BadStatusLine ) to be caught and collect few specific information for request sent,response recieved when that exception is caught in your next run? The total url with args and body to be dumped etc in a file?
3/ We are using requests library to make http calls under marvin, it seems there is a way to enable debug information for this library, check the below link. Please enable the debug information for this library and run the tests again to see by catching the complete request payload when this specific exception is raised.
http://docs.python-requests.org/en/master/api/#api-changes
# these two lines enable debugging at httplib level (requests->urllib3->httplib)
# you will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# the only thing missing will be the response.body which is not logged.
import httplib
httplib.HTTPConnection.debuglevel = 1
4/ One of the link suggests a trailing new line character for the request as the cause for this issue. Adding complete server calls and debug information will help to debug more i believe?
http://stackoverflow.com/questions/1767934/why-am-i-getting-this-error-in-python-httplib
Check above and you can get more information for this issue to delve further.
Thanks!
Santhosh
________________________________________
From: Girish Shilamkar [girish@clogeny.com]
Sent: Wednesday, October 16, 2013 3:01 AM
To: dev@cloudstack.apache.org
Subject: Re: Http connection failure in marvin
Lets track this issue here https://issues.apache.org/jira/browse/CLOUDSTACK-4846
Regards,
Girish
On 16-Oct-2013, at 11:30 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> httplib is used indirectly through requests. I'd try reproduce this
> by throttling 1000s of requests through a single mgmt server. It is
> intermittent because the mgmt server is probably loaded when you hit
> it and possibly producing an invalid/empty response.
Re: Http connection failure in marvin
Posted by Girish Shilamkar <gi...@clogeny.com>.
Lets track this issue here https://issues.apache.org/jira/browse/CLOUDSTACK-4846
Regards,
Girish
On 16-Oct-2013, at 11:30 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> httplib is used indirectly through requests. I'd try reproduce this
> by throttling 1000s of requests through a single mgmt server. It is
> intermittent because the mgmt server is probably loaded when you hit
> it and possibly producing an invalid/empty response.
Re: Http connection failure in marvin
Posted by Prasanna Santhanam <ts...@apache.org>.
On Wed, Oct 16, 2013 at 11:23:18AM +0530, Girish Shilamkar wrote:
> I have been hitting this issue quite often since couple of weeks.
> I just switched the management server and so far haven't seen it.
> Maybe updating httplib might help.
>
httplib is used indirectly through requests. I'd try reproduce this
by throttling 1000s of requests through a single mgmt server. It is
intermittent because the mgmt server is probably loaded when you hit
it and possibly producing an invalid/empty response.
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: Http connection failure in marvin
Posted by Girish Shilamkar <gi...@clogeny.com>.
I have been hitting this issue quite often since couple of weeks.
I just switched the management server and so far haven't seen it.
Maybe updating httplib might help.
Regards,
Girish
On 16-Oct-2013, at 11:04 AM, Prasanna Santhanam <ts...@apache.org> wrote:
> On Wed, Oct 16, 2013 at 05:11:46AM +0000, Girish Shilamkar wrote:
>> Hello,
>>
>> Sometimes while running automation testcases tests just fail with following error:
>>
>> ConnectionError: HTTPConnectionPool(host='10.147.59.65', port=8080):
>> Max retries exceeded with url: /client/api?
>> apiKey=lcgWQ9XrjyIyPjsl7Y0BGI2udSRMs6S1w_42lWkwpCSfLWjkgoWvCakWwUCJ3TqfgMmuHWT4DXv5Lgoz-KCuSQ&egressdefaultpolicy=true&serviceproviderlist%5B2%5D.
>> provider=VirtualRouter&serviceproviderlist%5B5%5D.service=Dhcp&displaytext=Network+offering-VR+services-8FM4XI&specifyVlan=False&serviceproviderlist%5B4%5D.
>> provider=VirtualRouter&serviceproviderlist%5B1%5D.provider=VirtualRouter&availability=Optional&conservemode=True&servicecapabilitylist%5B1%5D.
>> capabilitytype=SupportedSourceNatTypes&serviceproviderlist%5B3%5D.service=Dns&serviceproviderlist%5B5%5D.provider=VirtualRouter&response=json&serviceproviderlist%5B0%5D.
>> provider=VirtualRouter&servicecapabilitylist%5B0%5D.capabilitytype=RedundantRouter&serviceproviderlist%5B8%5D.provider=VirtualRouter&serviceproviderlist%5B1%5D.
>> service=Lb&servicecapabilitylist%5B0%5D.service=SourceNat&serviceproviderlist%5B4%5D.
>> service=Firewall&supportedservices=Dhcp%2CDns%2CSourceNat%2CPortForwarding%2CVpn%2CFirewall%2CLb%2CUserData%2CStaticNat&traffictype=GUEST&servicecapabilitylist%5B1%5D.
>> service=SourceNat&serviceproviderlist%5B6%5D.provider=VirtualRouter&serviceproviderlist%5B8%5D.service=StaticNat&serviceproviderlist%5B3%5D.provider=VirtualRouter&name=Network+
>> offering-VR+services-OLND5Q&guestiptype=Isolated&serviceproviderlist%5B7%5D.provider=VirtualRouter&servicecapabilitylist%5B1%5D.
>> capabilityvalue=peraccount&serviceproviderlist%5B7%5D.service=SourceNat&serviceproviderlist%5B2%5D.service=PortForwarding&servicecapabilitylist%5B0%5D.
>> capabilityvalue=true&serviceproviderlist%5B6%5D.service=Vpn&command=createNetworkOffering&signature=dFBPTQoc%2B%2B42%2FhDVyQQOFB87c7k%3D&serviceproviderlist%5B0%5D.service=UserData
>> (Caused by <class 'httplib.BadStatusLine'>: '')
>>
>
> We actually don't use the submitCmdsAndWait to send the requests to
> the mgmt server. We just send it through the apiclient and
> cloudConnection.py takes over. I've seen the BadStatusLine issue with
> httplib on our test systems too. It is pretty random. At first I
> thought it was the throttling of the API but we don't enable that
> setting. I'm still looking for why this happens ... and how to repro
> it.
>
>
>> These errors are intermittent. While investigating this problem I found this in marvin.cloudstackTestClient
>>
>> def submitCmdsAndWait(self, cmds, workers=1):
>> '''FixME, httplib has issue if more than one thread submitted'''
>> if self.asyncJobMgr is None:
>> self.asyncJobMgr = asyncJobMgr.asyncJobMgr(self.apiClient,
>> self.dbConnection)
>> return self.asyncJobMgr.submitCmdsAndWait(cmds, workers)
>>
>> My theory is if the response from management server slow, multiple thread are submitted and we hit this issue.
>>
>> Please advise.
>>
>> Regards,
>> Girish
>>
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
Re: Http connection failure in marvin
Posted by Prasanna Santhanam <ts...@apache.org>.
On Wed, Oct 16, 2013 at 05:11:46AM +0000, Girish Shilamkar wrote:
> Hello,
>
> Sometimes while running automation testcases tests just fail with following error:
>
> ConnectionError: HTTPConnectionPool(host='10.147.59.65', port=8080):
> Max retries exceeded with url: /client/api?
> apiKey=lcgWQ9XrjyIyPjsl7Y0BGI2udSRMs6S1w_42lWkwpCSfLWjkgoWvCakWwUCJ3TqfgMmuHWT4DXv5Lgoz-KCuSQ&egressdefaultpolicy=true&serviceproviderlist%5B2%5D.
> provider=VirtualRouter&serviceproviderlist%5B5%5D.service=Dhcp&displaytext=Network+offering-VR+services-8FM4XI&specifyVlan=False&serviceproviderlist%5B4%5D.
> provider=VirtualRouter&serviceproviderlist%5B1%5D.provider=VirtualRouter&availability=Optional&conservemode=True&servicecapabilitylist%5B1%5D.
> capabilitytype=SupportedSourceNatTypes&serviceproviderlist%5B3%5D.service=Dns&serviceproviderlist%5B5%5D.provider=VirtualRouter&response=json&serviceproviderlist%5B0%5D.
> provider=VirtualRouter&servicecapabilitylist%5B0%5D.capabilitytype=RedundantRouter&serviceproviderlist%5B8%5D.provider=VirtualRouter&serviceproviderlist%5B1%5D.
> service=Lb&servicecapabilitylist%5B0%5D.service=SourceNat&serviceproviderlist%5B4%5D.
> service=Firewall&supportedservices=Dhcp%2CDns%2CSourceNat%2CPortForwarding%2CVpn%2CFirewall%2CLb%2CUserData%2CStaticNat&traffictype=GUEST&servicecapabilitylist%5B1%5D.
> service=SourceNat&serviceproviderlist%5B6%5D.provider=VirtualRouter&serviceproviderlist%5B8%5D.service=StaticNat&serviceproviderlist%5B3%5D.provider=VirtualRouter&name=Network+
> offering-VR+services-OLND5Q&guestiptype=Isolated&serviceproviderlist%5B7%5D.provider=VirtualRouter&servicecapabilitylist%5B1%5D.
> capabilityvalue=peraccount&serviceproviderlist%5B7%5D.service=SourceNat&serviceproviderlist%5B2%5D.service=PortForwarding&servicecapabilitylist%5B0%5D.
> capabilityvalue=true&serviceproviderlist%5B6%5D.service=Vpn&command=createNetworkOffering&signature=dFBPTQoc%2B%2B42%2FhDVyQQOFB87c7k%3D&serviceproviderlist%5B0%5D.service=UserData
> (Caused by <class 'httplib.BadStatusLine'>: '')
>
We actually don't use the submitCmdsAndWait to send the requests to
the mgmt server. We just send it through the apiclient and
cloudConnection.py takes over. I've seen the BadStatusLine issue with
httplib on our test systems too. It is pretty random. At first I
thought it was the throttling of the API but we don't enable that
setting. I'm still looking for why this happens ... and how to repro
it.
> These errors are intermittent. While investigating this problem I found this in marvin.cloudstackTestClient
>
> def submitCmdsAndWait(self, cmds, workers=1):
> '''FixME, httplib has issue if more than one thread submitted'''
> if self.asyncJobMgr is None:
> self.asyncJobMgr = asyncJobMgr.asyncJobMgr(self.apiClient,
> self.dbConnection)
> return self.asyncJobMgr.submitCmdsAndWait(cmds, workers)
>
> My theory is if the response from management server slow, multiple thread are submitted and we hit this issue.
>
> Please advise.
>
> Regards,
> Girish
>
--
Prasanna.,
------------------------
Powered by BigRock.com