You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@libcloud.apache.org by Noel Milton Vega <nm...@computingarchitects.com> on 2012/06/22 01:30:32 UTC
Re: Terremark authentication example using libcloud ...
Hi Sengork (et. al):
It's been longer than I anticipated in getting back to this issue
and thread (... apologies for going dark while I tended to other
matters).
Continuing where we left off, thank you for the below feedback.
In troubleshooting, I believe the issue lies in the below code
within "vcloud.py". vcloud.py is the same for python2 & python3,
but the below code (and perhaps other "shared code" too)
behaves slightly differently between interpreters:
=================================
vcloud.py code snip ...
=================================
def _get_auth_headers(self):
"""Some providers need different headers than others"""
return {
'Authorization':
"Basic %s"
% base64.b64encode(b('%s:%s' % (self.user_id, self.key))),
'Content-Length': 0,
'Accept': 'application/*+xml'
}
=================================
Specifically, the "base64.b64encode ..." line returns the following
for each interpreter... Notice the byte-string for the python-3 case:
PYTHON 2 (Works):
curl -i -X POST -H 'Content-Length: 0' -H 'X-LC-Request-ID: 31763344' -H 'Accept: application/*+xml'
-H 'Authorization: Basic xxxxxyyyyyzzzzzz'
--compress https://services.vcloudexpress.terremark.com:443/api/v0.8/login#
python 3 (Fails):
curl -i -X POST -H 'Content-Length: 0' -H 'X-LC-Request-ID: 31862928' -H 'Accept: application/*+xml'
-H 'Authorization: Basic b'"'"'xxxxxyyyyyyyzzzzzz'"'"''
--compress https://services.vcloudexpress.terremark.com:443/api/v0.8/login
When I temporarily substituted my auth string in the above code directly,
it worked for python 3 too.
=========================
Three things to mention:
=========================
(1) I was surprised that that base64 line worked at all (even in Python 2)
since I always thought use of "b" only worked for string literals
(not generated strings).
(2) Not an issue for this specific thread, but even when I hard-coded
my auth string (just to get past the Python-3 related auth issue), sadly
another unrelated (xml parsing) issue arose almost immediately.
Just as a heads up, it was this:
Traceback (most recent call last):
File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1668, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 1
(So I never got my list_images() output.)
(3) Similar to the last point, python-2 gets much further along, but also
eventually experiences an parse-related exception too (as it's
iterating through the list the of images too I believe). Again, as a heads-up fyi:
File "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 76, in __init__
raise Exception(self.parse_error())
Exception: <Element '{http://www.w3.org/1999/xhtml}html' at 0x118dd10>
Thanks,
Noel
Hi Noel, That narrows down the issue to obtaining an authentication token (via
_get_auth_token) which is the very first thing libcloud vCloud driver
needs to do before it can issue subsequent HTTP calls. Note this is
the only time when your base64 encoded "user:pass" will be sent in a
HTTP request. What is failing is the fact API web service is not
accepting your credentials and replying with an authentication token
to be used in subsequent HTTP requests. Couple of things to consider:
- Is the API username equal to the full email address?
- Try reproducing the failure using Python 2.6/2.7
- Verify your base64 encoded string in curl request body matches your
"username:password"
import base64
base64.b64decode("auth string")
- Comment out this section of _get_auth_headers from vcloud.py "
'Accept': 'application/*+xml' " and retry authenticating
- HTTP call is doing the right thing according to Terremark's v0.8 API
doc (a POST to the right URL with right base64 encoding), however I
have noticed Terremark's developer forum has a few older threads
describing similar API auth issues which were fixed by support staff.
Might be worth double checking there is no active issue at the API
end. There was no Providers.VCLOUD until we've introduced vCloud v1.5
support (which is only in trunk right now). Looks like you should not need VERIFY_SSL_CERT = False as Terremark
seem to have a signed SSL cert @
https://services.vcloudexpress.terremark.com For reference here's the full request/response body I receive for a
dummy "user:pass" see how it compares: # -------- begin 47775432 request ----------
curl -i -X POST -H 'Content-Length: 0' -H 'X-LC-Request-ID: 47775432'
-H 'Authorization: Basic dXNlcjpwYXNz' https://services.vcloudexpress.terremark.com:443/api/v0.8/login# -------- begin 47775432:47788872 response ----------
HTTP/1.1 401 Unauthorized
Content-Length: 1293
X-Powered-By: ASP.NET Server: Microsoft-IIS/7.0
Date: Sat, 12 May 2012 02:26:10 GMT
Content-Type: text/html
Www-Authenticate: Basic Realm="vCloud Api" <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";>
<html xmlns="http://www.w3.org/1999/xhtml";>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>401 - Unauthorized: Access is denied due to invalid credentials.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica,
sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px
2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
<div class="content-container"><fieldset>
<h2>401 - Unauthorized: Access is denied due to invalid credentials.</h2>
<h3>You do not have permission to view this directory or page using
the credentials that you supplied.</h3>
</fieldset></div>
</div>
</body>
</html> On 12 May 2012 05:03, Noel Milton Vega <nm...@computingarchitects.com> wrote:
>
> PS... Ignore my comment about "conn.list_sizes()" in the below
> code not generating a HTTP call. I looked at the base.py source
> and see that my usage (of the returned class) below is slightly
> off. :)
>
> My original auth problem below still remains. I continue to
> investigate it.
>
> Thanks again,
> -- nmv
>
>
>
>
> >________________________________
> > From: Noel Milton Vega <nm...@computingarchitects.com>
> >To: "users@libcloud.apache.org" <us...@libcloud.apache.org>
> >Sent: Friday, May 11, 2012 2:25 PM
> >Subject: Re: Terremark authentication example using libcloud ...
> >
> >Hello Sengork:
> >
> >Thank you for the reply. I'll formulate this reply in "PARTS" for
> >clarity.
> >
> >=============
> >
> >PART1:
> >=============
> >
> >Thanks for the gentle reminder to include the libcloud
> >version, which I should have originally stated is
> >version "0.9.1" for python3. Obtained via: >>> libcloud.__version__
> >
> >
> >=============
> >
> >PART2:
> >=============
> >
> >Per your example below, I tried running it again, this time by
> >inserting the following prior to the get_driver() call:
> >
> >
> > libcloud.security.VERIFY_SSL_CERT = False
> >
> >Sadly, it did not resolve the issue. Here again, for referential
> >convenience, is the cleaned up code (including my
> >in-line comments where the exception is being raised):
> >
> >> ###############################################
> >> #! /usr/bin/env python3
> >>
> >> import libcloud.compute.types
> >> import libcloud.compute.providers
> >> import libcloud.security
> >> #
> >> TERREMARK_USERNAME = "some@email.address"
> >> TERREMARK_PASSWORD = "some-password"
> >> #
> >> libcloud.security.VERIFY_SSL_CERT = False
> >> # Including this or not doesn't change the exception outcome.
> >
> >> Driver = libcloud.compute.providers.get_driver(libcloud.compute.types.Provider.TERREMARK)
> >> conn = Driver(TERREMARK_USERNAME, TERREMARK_PASSWORD)
> >>
> >
> >> for aSize in conn.list_sizes(): print(aSize) # No HTTP request generated here.
Why? Hmmm.
> >
> >> # Oddly, this call to list_sizes() doesn't seem to generate a HTTP
request (i.e. no curl(1)
> >> # output for this). However it does return a Python list with seemingly
fake/dummy VM
> >> # sizes generated internally by libcloud.
> >
> >>
> >> for anImage in conn.list_images(): print(anImage) # HTTP request is generated
here, but so is the exception.
> >> # On the other hand, this call to list_images() does generate a HTTP
request (and curl(1) output),
> >> # and causes the exception above to be raised. See below (in this
email) for a cut/paste of
> >> # curl(1) output as well as related python exception.
> >
> >>
> >> [ ... snip ... ]
> >> #########################################
> >The exception, with or without the VERIFY_SSL_CERT flag, is the same and
> >appears as follows:
> >
> >-------------------
> >IN CURL(1) LOG:
> >
> >-------------------
> >
> > # -------- begin 27800016 request ----------
> > curl -i -X POST -H 'Content-Length: 0' -H 'X-LC-Request-ID: 27800016'
> > -H 'Authorization: Basic b'"'"'a-60-character-scring'"'"''
> > --compress https://services.vcloudexpress.terremark.com:443/api/v0.8/login
> > # -------- begin 27800016:27800208 response ----------
> > HTTP/1.1 401 Unauthorized
> > Content-Type: text/html
> > Server: Microsoft-IIS/7.0
> > Www-Authenticate: Basic Realm="vCloud Api"
> > X-Powered-By: ASP.NET
> > Date: Fri, 11 May 2012 17:31:52 GMT
> > Content-Length: 1293
> > [ ... snip ... ]
> >
> >
> >--------------------
> >
> >IN Python3 IDE:
> >--------------------
> >
> > xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column
1
> > File "/home/nmvega/Dropbox/CodeDEV.d/PYTHON.d/Libcloud-Client-Development.d/terremark.py",
line 28, in <module>
> > for anImage in conn.list_images(): print(anImage)
> > File "/usr/lib/python3.2/site-packages/libcloud/compute/drivers/vcloud.py",
line 474, in list_images
> > for vdc in self.vdcs:
> > File "/usr/lib/python3.2/site-packages/libcloud/compute/drivers/vcloud.py",
line 281, in vdcs
> > self.connection.check_org() # make sure the org is set.
> > File "/usr/lib/python3.2/site-packages/libcloud/compute/drivers/vcloud.py",
line 226, in check_org
> > self._get_auth_token()
> > File "/usr/lib/python3.2/site-packages/libcloud/compute/drivers/vcloud.py",
line 246, in _get_auth_token
> > body = ET.XML(resp.read())
> > File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1329, in XML
> > parser.feed(text)
> > File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1662, in feed
> >
> >
> >
> >
> >=================
> >
> >PART3:
> >=================
> >Finally, just a note that in v0.9.1 of libcloud, the source file
> >/usr/lib/python3.2/site-packages/libcloud/compute/providers.py
> >doesn't have a dict() entry for Providers.VCLOUD, so I couldn't try
> >your included example (... Just for grins -- it would not have worked
> >for TERREMARK anyay, since they use a modified version of the
> >vCloud API). Just a side note that Providers.VCLOUD was not there.
> >
> >
> >Thank you in advance again Sengork, et. al.
> >
> >
> >-- nmv
> >
> >
> >
> >
> >>________________________________
> >> From: Sengor <se...@gmail.com>
> >>To: users@libcloud.apache.org; Noel Milton Vega <nm...@computingarchitects.com>
> >>Sent: Thursday, May 10, 2012 11:37 PM
> >>Subject: Re: Terremark authentication example using libcloud ...
> >>
> >>Hi,
> >>
> >>I've not used the Terremark variant of vCloud driver, but this is what
> >>should work:
> >>conn = Driver(TERREMARK_USERNAME, TERREMARK_PASSWORD)
> >>
> >>The driver will perform base64 encoding and joining "user:name" on your
> >>behalf.
> >>
> >>As an example this is what's done for vCloud v1.5:
> >>
> >>from libcloud.compute.types import Provider
> >>from libcloud.compute.providers import get_driver
> >># Do this only if API has self signed SSL certificate
> >>import libcloud.security
> >>libcloud.security.VERIFY_SSL_CERT = False
> >>Driver = get_driver(Provider.VCLOUD)
> >>conn = Driver("user@organisation", "password", host="hostname.com",
> >>api_version="1.5")
> >>conn.list_images()
> >>
> >>You should not need api_version and host for v0.8.
> >>
> >>I suggest that you enable libcloud debugging and observe the raw API
> >>request/responses for any hints as to the cause of the issue.
> >>
> >>$ export LIBCLOUD_DEBUG=/tmp/raw
> >>$ tail -f /tmp/raw
> >>$ python
> >>.....
> >>
> >>Please also let us know which version of libcloud are you using?
> >>
> >>
> >>
> >>
> >>On 11 May 2012 06:00, Noel Milton Vega <nm...@computingarchitects.com>wrote:
> >>
> >>> Hello friends:
> >>>
> >>> I was wondering if anyone had a Terremark libcloud authentication
> >>> example, similar to the ones indicated here for Amazon EC2 and Rackspace:
> >>>
> >>> http://libcloud.apache.org/getting-started.html
> >>>
> >>> I've tried a few variations of the examples provided in the above link for
> >>> the Terremark case (a vCloud API implementation), but received an "Invalid
> >>> Credentials with the provider" Exception in each case.
> >>>
> >>> So far I've tried passing Authentication using the three (3) variations
> >>> shown below (commented out) for "conn = Driver()".
> >>>
> >>> Thank you in advance!
> >>>
> >>> ###############################################
> >>> #! /usr/bin/env python3
> >>>
> >>> import libcloud.compute.types
> >>> import libcloud.compute.providers
> >>> import base64
> >>> #
> >>> TERREMARK_USERNAME = "some@email.address"
> >>> TERREMARK_PASSWORD = "some-password"
> >>> TERREMARK_B64AUTH = base64.b64encode(bytes(TERREMARK_USERNAME + ":" +
> >>> TERREMARK_PASSWORD, encoding='utf-8'))
> >>> #
> >>> Driver =
> >>> libcloud.compute.providers.get_driver(libcloud.compute.types.Provider.TERREMARK)
> >>>
> >>> #conn = Driver(TERREMARK_USERNAME, TERREMARK_PASSWORD)
> >>> #conn = Driver(TERREMARK_B64AUTH)
> >>> #conn = Driver(TERREMARK_USERNAME + ":" + TERREMARK_PASSWORD)
> >>>
> >>> [ ... snip ... ]
> >>> #########################################
> >>>
> >>
> >>
> >>
> >>--
> >>sengork
> >>
> >>
> >>
> >
> > --
sengork
Re: Terremark authentication example using libcloud ...
Posted by Sengor <se...@gmail.com>.
Hi Noel,
One thing to try could perhaps be the vcloud driver which predates
vCloud 1.5 overhaul. This is due to the fact we modified a couple of
shared chunks of code between non-1.5 and 1.5 versions of the vcloud
driver. This could indicate whether it's a regression problem we were
unable to test out or the Terremark service no longer works with the
original driver version (assuming it worked before - which it should
have).
Try revision 1210045 of vcloud.py with Python 2.7.x and see if it
behaves the same way:
https://svn.apache.org/viewvc/libcloud/trunk/libcloud/compute/drivers/vcloud.py?view=log
That revision is just before we started modifications for vCloud 1.5.
I hope this helps us progress further.
On 26 June 2012 08:59, Noel Milton Vega <nm...@computingarchitects.com> wrote:
> Hi Sengor:
>
> Thanks for the replies.
>
> I manually patched vcloud.py by appending ".decide('utf-8')" in two
> the places needing it, and that worked (... and makes sense ... I had
> to the same decode() step in my non-libcloud httplib2/ReST case for
> python-3).
>
> So let me move onto the subsequent parsing exceptions I mentioned...
> which is raised during processing of the HTTP XML Response
> (indicating malformed). Tracking it down, this happens in
> vcloud.py, here:
>
> =================
> class VCloudConnection(ConnectionUserAndKey):
> <snip>
>
> def _get_auth_token(self):
> ...
> <snip>
> body = ET.XML(resp.read()) <---
> <snip>
> ...
> <snip>
> =================
>
>
> Through various print() and type() statements, I see that "resp.read()", which
> when expanded translates to "http.client.HTTPResponse.read()", returns a
> bytes string. However, I'm wondering if this may also need to be similarly
> ".decode('utf-8')", like this:
>
> body = ET.XML(resp.read().decode('utf-8'))
>
>
> since the Python-3 documentation for "xml.etree.ElementTree.XML()"
> (which is what ET.XML() expands to), indicates that it is expecting a string
> (not a bytes encoded string).
>
> So I tried the modification above. Here is the before and after:
>
> resp.read() [ BEFORE .decode() inserted ]
>
> <class 'bytes'> <--- types is 'bytes', and the following is the actual bytes string...
>
> b'b\'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\\n <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\\n</OrgL'
>
>
> resp.read().decode('utf-8') [ AFTER .decode() inserted ]
>
>
> <class 'str'> <--- types is 'str', and the following is the actual string...
>
> b'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\n <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\n</OrgL
>
>
> Looking carefully at both the bytes string (un-decoded) and the string (decoded) above -- forgetting about whether we
> decode() it or not for the moment -- both sequences appear to be malformed... They both have an extra "b'"
> pre-pending it, and both are truncated at the end (i.e. the enclosing "</OrgL" ... doesn't end correctly). Note: Just
>
> in case this was a print() artefact, I also wrote the output to a file (same truncation issue).
>
> This above issue is only true for python-3. Python 2, (decode()ed or not) emits output correctly as:
>
> <OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/></OrgList>
>
>
> So there appears to be two problems here:
> (1) missing ".decode('utf-8')" method call.
> (2) the output of "resp.read()", with or without the ".decode('utf-8')" part, appears to be malformed.
>
>
> To make matters more interesting, note that the python-3 case curl(1) output shows a result that is not truncated,
> but still, however, has the pre-pending "b'" (so it's still malformed). And here is that curl(1) output
> (remembering that the above were from print() debug statements of mine, not from curl(1)):
>
> b'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\n <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\n</OrgList>'
>
>
> As I look more for the cause, any helpful feedback is appreciated.
>
>
> Again,... the libcloud API is v0.10.1 with Python v3.2.3
>
>
> Thank you,
> Noel
>
>
>>________________________________
>> From: Sengor <se...@gmail.com>
>>To: users@libcloud.apache.org; Noel Milton Vega <nm...@computingarchitects.com>
>>Sent: Thursday, June 21, 2012 10:21 PM
>>Subject: Re: Terremark authentication example using libcloud ...
>>
>>Hi Noel,
>>
>>Replies in line below:
>>
>>On 22 June 2012 09:30, Noel Milton Vega <nm...@computingarchitects.com> wrote:
>>> (1) I was surprised that that base64 line worked at all (even in Python 2)
>>> since I always thought use of "b" only worked for string literals
>>> (not generated strings).
>>
>>Patch has been applied to trunk which now works for both Python 2 & 3.
>>For more info see: https://issues.apache.org/jira/browse/LIBCLOUD-204
>>
>>
>>> (2) Not an issue for this specific thread, but even when I hard-coded
>>> my auth string (just to get past the Python-3 related auth issue), sadly
>>> another unrelated (xml parsing) issue arose almost immediately.
>>> Just as a heads up, it was this:
>>>
>>>
>>> Traceback (most recent call last):
>>> File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1668, in feed
>>> self._parser.Parse(data, 0)
>>> xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 1
>>>
>>> (So I never got my list_images() output.)
>>>
>>> (3) Similar to the last point, python-2 gets much further along, but also
>>> eventually experiences an parse-related exception too (as it's
>>> iterating through the list the of images too I believe). Again, as a heads-up fyi:
>>>
>>>
>>> File "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 76, in __init__
>>> raise Exception(self.parse_error())
>>> Exception: <Element '{http://www.w3.org/1999/xhtml}html'; at 0x118dd10>
>>
>>Hard to say what the issue could be for these two. Could you post the
>>debugging output for these two (don't forget to remove the private
>>contents such as base64 encoded auth):
>>http://libcloud.apache.org/docs/debugging.html
>>
>>Note I've only used the vCloud module under Python 2.x at the moment
>>and don't have access to Terremark. We did slightly change the VDC
>>object model when vCloud v1.5 support has been introduced. If this is
>>related to what you see it'd be good to fix it up, otherwise it's
>>something else getting in the way.
>>
>>--
>>sengork
>>
>>
>>
--
sengork
Re: Terremark authentication example using libcloud ...
Posted by Noel Milton Vega <nm...@computingarchitects.com>.
Oops... Disregard the last part of my email below -- my comment
about the the curl(1) output.
The curl(1) output shows a properly formatted bytes string returned in the
response. But, however, properly formatted that curl(1) output may be,
the problem is still as I described below (and we are not dealing with
curl(1) output, but python call output). Thanks.
----- Original Message -----
> From: Noel Milton Vega <nm...@computingarchitects.com>
> To: "users@libcloud.apache.org" <us...@libcloud.apache.org>
> Cc:
> Sent: Monday, June 25, 2012 6:59 PM
> Subject: Re: Terremark authentication example using libcloud ...
>
> Hi Sengor:
>
> Thanks for the replies.
>
> I manually patched vcloud.py by appending ".decide('utf-8')"
> in two
> the places needing it, and that worked (... and makes sense ... I had
> to the same decode() step in my non-libcloud httplib2/ReST case for
> python-3).
>
> So let me move onto the subsequent parsing exceptions I mentioned...
> which is raised during processing of the HTTP XML Response
> (indicating malformed). Tracking it down, this happens in
> vcloud.py, here:
>
> =================
> class VCloudConnection(ConnectionUserAndKey):
> <snip>
>
> def _get_auth_token(self):
> ...
> <snip>
> body = ET.XML(resp.read()) <---
> <snip>
> ...
> <snip>
> =================
>
>
> Through various print() and type() statements, I see that
> "resp.read()", which
> when expanded translates to "http.client.HTTPResponse.read()", returns
> a
> bytes string. However, I'm wondering if this may also need to be similarly
> ".decode('utf-8')", like this:
>
> body = ET.XML(resp.read().decode('utf-8'))
>
>
> since the Python-3 documentation for "xml.etree.ElementTree.XML()"
> (which is what ET.XML() expands to), indicates that it is expecting a string
> (not a bytes encoded string).
>
> So I tried the modification above. Here is the before and after:
>
> resp.read() [ BEFORE .decode() inserted ]
>
> <class 'bytes'> <--- types is 'bytes', and the
> following is the actual bytes string...
>
> b'b\'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:xsd="http://www.w3.org/2001/XMLSchema">\\n <Org
> href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx"
> type="application/vnd.vmware.vcloud.org+xml"
> name="email@example.com"/>\\n</OrgL'
>
>
> resp.read().decode('utf-8') [ AFTER .decode() inserted ]
>
>
> <class 'str'> <--- types is 'str', and the following
> is the actual string...
>
> b'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:xsd="http://www.w3.org/2001/XMLSchema">\n <Org
> href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx"
> type="application/vnd.vmware.vcloud.org+xml"
> name="email@example.com"/>\n</OrgL
>
>
> Looking carefully at both the bytes string (un-decoded) and the string (decoded)
> above -- forgetting about whether we
> decode() it or not for the moment -- both sequences appear to be malformed...
> They both have an extra "b'"
> pre-pending it, and both are truncated at the end (i.e. the enclosing
> "</OrgL" ... doesn't end correctly). Note: Just
>
> in case this was a print() artefact, I also wrote the output to a file (same
> truncation issue).
>
> This above issue is only true for python-3. Python 2, (decode()ed or not) emits
> output correctly as:
>
> <OrgList xmlns="http://www.vmware.com/vcloud/v0.8"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <Org
> href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx"
> type="application/vnd.vmware.vcloud.org+xml"
> name="email@example.com"/></OrgList>
>
>
> So there appears to be two problems here:
> (1) missing ".decode('utf-8')" method call.
> (2) the output of "resp.read()", with or without the
> ".decode('utf-8')" part, appears to be malformed.
>
>
> To make matters more interesting, note that the python-3 case curl(1) output
> shows a result that is not truncated,
> but still, however, has the pre-pending "b'" (so it's still
> malformed). And here is that curl(1) output
> (remembering that the above were from print() debug statements of mine, not from
> curl(1)):
>
> b'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:xsd="http://www.w3.org/2001/XMLSchema">\n <Org
> href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx"
> type="application/vnd.vmware.vcloud.org+xml"
> name="email@example.com"/>\n</OrgList>'
>
>
> As I look more for the cause, any helpful feedback is appreciated.
>
>
> Again,... the libcloud API is v0.10.1 with Python v3.2.3
>
>
> Thank you,
> Noel
>
>
>> ________________________________
>> From: Sengor <se...@gmail.com>
>> To: users@libcloud.apache.org; Noel Milton Vega
> <nm...@computingarchitects.com>
>> Sent: Thursday, June 21, 2012 10:21 PM
>> Subject: Re: Terremark authentication example using libcloud ...
>>
>> Hi Noel,
>>
>> Replies in line below:
>>
>> On 22 June 2012 09:30, Noel Milton Vega
> <nm...@computingarchitects.com> wrote:
>>> (1) I was surprised that that base64 line worked at all (even in Python
> 2)
>>> since I always thought use of "b" only worked for string
> literals
>>> (not generated strings).
>>
>> Patch has been applied to trunk which now works for both Python 2 & 3.
>> For more info see: https://issues.apache.org/jira/browse/LIBCLOUD-204
>>
>>
>>> (2) Not an issue for this specific thread, but even when I hard-coded
>>> my auth string (just to get past the Python-3 related auth issue),
> sadly
>>> another unrelated (xml parsing) issue arose almost immediately.
>>> Just as a heads up, it was this:
>>>
>>>
>>> Traceback (most recent call last):
>>> File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line
> 1668, in feed
>>> self._parser.Parse(data, 0)
>>> xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1,
> column 1
>>>
>>> (So I never got my list_images() output.)
>>>
>>> (3) Similar to the last point, python-2 gets much further along, but
> also
>>> eventually experiences an parse-related exception too (as it's
>>> iterating through the list the of images too I believe). Again, as
> a heads-up fyi:
>>>
>>>
>>> File
> "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 76,
> in __init__
>>> raise Exception(self.parse_error())
>>> Exception: <Element '{http://www.w3.org/1999/xhtml}html'; at
> 0x118dd10>
>>
>> Hard to say what the issue could be for these two. Could you post the
>> debugging output for these two (don't forget to remove the private
>> contents such as base64 encoded auth):
>> http://libcloud.apache.org/docs/debugging.html
>>
>> Note I've only used the vCloud module under Python 2.x at the moment
>> and don't have access to Terremark. We did slightly change the VDC
>> object model when vCloud v1.5 support has been introduced. If this is
>> related to what you see it'd be good to fix it up, otherwise it's
>> something else getting in the way.
>>
>> --
>> sengork
>>
>>
>>
>
Re: Terremark authentication example using libcloud ...
Posted by Noel Milton Vega <nm...@computingarchitects.com>.
Hi Sengor:
Thanks for the replies.
I manually patched vcloud.py by appending ".decide('utf-8')" in two
the places needing it, and that worked (... and makes sense ... I had
to the same decode() step in my non-libcloud httplib2/ReST case for
python-3).
So let me move onto the subsequent parsing exceptions I mentioned...
which is raised during processing of the HTTP XML Response
(indicating malformed). Tracking it down, this happens in
vcloud.py, here:
=================
class VCloudConnection(ConnectionUserAndKey):
<snip>
def _get_auth_token(self):
...
<snip>
body = ET.XML(resp.read()) <---
<snip>
...
<snip>
=================
Through various print() and type() statements, I see that "resp.read()", which
when expanded translates to "http.client.HTTPResponse.read()", returns a
bytes string. However, I'm wondering if this may also need to be similarly
".decode('utf-8')", like this:
body = ET.XML(resp.read().decode('utf-8'))
since the Python-3 documentation for "xml.etree.ElementTree.XML()"
(which is what ET.XML() expands to), indicates that it is expecting a string
(not a bytes encoded string).
So I tried the modification above. Here is the before and after:
resp.read() [ BEFORE .decode() inserted ]
<class 'bytes'> <--- types is 'bytes', and the following is the actual bytes string...
b'b\'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\\n <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\\n</OrgL'
resp.read().decode('utf-8') [ AFTER .decode() inserted ]
<class 'str'> <--- types is 'str', and the following is the actual string...
b'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\n <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\n</OrgL
Looking carefully at both the bytes string (un-decoded) and the string (decoded) above -- forgetting about whether we
decode() it or not for the moment -- both sequences appear to be malformed... They both have an extra "b'"
pre-pending it, and both are truncated at the end (i.e. the enclosing "</OrgL" ... doesn't end correctly). Note: Just
in case this was a print() artefact, I also wrote the output to a file (same truncation issue).
This above issue is only true for python-3. Python 2, (decode()ed or not) emits output correctly as:
<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/></OrgList>
So there appears to be two problems here:
(1) missing ".decode('utf-8')" method call.
(2) the output of "resp.read()", with or without the ".decode('utf-8')" part, appears to be malformed.
To make matters more interesting, note that the python-3 case curl(1) output shows a result that is not truncated,
but still, however, has the pre-pending "b'" (so it's still malformed). And here is that curl(1) output
(remembering that the above were from print() debug statements of mine, not from curl(1)):
b'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\n <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\n</OrgList>'
As I look more for the cause, any helpful feedback is appreciated.
Again,... the libcloud API is v0.10.1 with Python v3.2.3
Thank you,
Noel
>________________________________
> From: Sengor <se...@gmail.com>
>To: users@libcloud.apache.org; Noel Milton Vega <nm...@computingarchitects.com>
>Sent: Thursday, June 21, 2012 10:21 PM
>Subject: Re: Terremark authentication example using libcloud ...
>
>Hi Noel,
>
>Replies in line below:
>
>On 22 June 2012 09:30, Noel Milton Vega <nm...@computingarchitects.com> wrote:
>> (1) I was surprised that that base64 line worked at all (even in Python 2)
>> since I always thought use of "b" only worked for string literals
>> (not generated strings).
>
>Patch has been applied to trunk which now works for both Python 2 & 3.
>For more info see: https://issues.apache.org/jira/browse/LIBCLOUD-204
>
>
>> (2) Not an issue for this specific thread, but even when I hard-coded
>> my auth string (just to get past the Python-3 related auth issue), sadly
>> another unrelated (xml parsing) issue arose almost immediately.
>> Just as a heads up, it was this:
>>
>>
>> Traceback (most recent call last):
>> File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1668, in feed
>> self._parser.Parse(data, 0)
>> xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 1
>>
>> (So I never got my list_images() output.)
>>
>> (3) Similar to the last point, python-2 gets much further along, but also
>> eventually experiences an parse-related exception too (as it's
>> iterating through the list the of images too I believe). Again, as a heads-up fyi:
>>
>>
>> File "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 76, in __init__
>> raise Exception(self.parse_error())
>> Exception: <Element '{http://www.w3.org/1999/xhtml}html'; at 0x118dd10>
>
>Hard to say what the issue could be for these two. Could you post the
>debugging output for these two (don't forget to remove the private
>contents such as base64 encoded auth):
>http://libcloud.apache.org/docs/debugging.html
>
>Note I've only used the vCloud module under Python 2.x at the moment
>and don't have access to Terremark. We did slightly change the VDC
>object model when vCloud v1.5 support has been introduced. If this is
>related to what you see it'd be good to fix it up, otherwise it's
>something else getting in the way.
>
>--
>sengork
>
>
>
Re: Terremark authentication example using libcloud ...
Posted by Sengor <se...@gmail.com>.
Hi Noel,
Replies in line below:
On 22 June 2012 09:30, Noel Milton Vega <nm...@computingarchitects.com> wrote:
> (1) I was surprised that that base64 line worked at all (even in Python 2)
> since I always thought use of "b" only worked for string literals
> (not generated strings).
Patch has been applied to trunk which now works for both Python 2 & 3.
For more info see: https://issues.apache.org/jira/browse/LIBCLOUD-204
> (2) Not an issue for this specific thread, but even when I hard-coded
> my auth string (just to get past the Python-3 related auth issue), sadly
> another unrelated (xml parsing) issue arose almost immediately.
> Just as a heads up, it was this:
>
>
> Traceback (most recent call last):
> File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1668, in feed
> self._parser.Parse(data, 0)
> xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 1
>
> (So I never got my list_images() output.)
>
> (3) Similar to the last point, python-2 gets much further along, but also
> eventually experiences an parse-related exception too (as it's
> iterating through the list the of images too I believe). Again, as a heads-up fyi:
>
>
> File "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 76, in __init__
> raise Exception(self.parse_error())
> Exception: <Element '{http://www.w3.org/1999/xhtml}html' at 0x118dd10>
Hard to say what the issue could be for these two. Could you post the
debugging output for these two (don't forget to remove the private
contents such as base64 encoded auth):
http://libcloud.apache.org/docs/debugging.html
Note I've only used the vCloud module under Python 2.x at the moment
and don't have access to Terremark. We did slightly change the VDC
object model when vCloud v1.5 support has been introduced. If this is
related to what you see it'd be good to fix it up, otherwise it's
something else getting in the way.
--
sengork