Re: Terremark authentication example using libcloud ...

[Sengor <se...@gmail.com>]

Hi Noel,

One thing to try could perhaps be the vcloud driver which predates
vCloud 1.5 overhaul. This is due to the fact we modified a couple of
shared chunks of code between non-1.5 and 1.5 versions of the vcloud
driver. This could indicate whether it's a regression problem we were
unable to test out or the Terremark service no longer works with the
original driver version (assuming it worked before - which it should
have).

Try revision 1210045 of vcloud.py with Python 2.7.x and see if it
behaves the same way:
https://svn.apache.org/viewvc/libcloud/trunk/libcloud/compute/drivers/vcloud.py?view=log

That revision is just before we started modifications for vCloud 1.5.
I hope this helps us progress further.


On 26 June 2012 08:59, Noel Milton Vega <nm...@computingarchitects.com> wrote:
> Hi Sengor:
>
> Thanks for the replies.
>
> I manually patched vcloud.py by appending ".decide('utf-8')" in two
> the places needing it, and that worked (... and makes sense ... I had
> to the same decode() step in my non-libcloud httplib2/ReST case for
> python-3).
>
> So let me move onto the subsequent parsing exceptions I mentioned...
> which is raised during processing of the HTTP XML Response
> (indicating malformed). Tracking it down, this happens in
> vcloud.py, here:
>
>   =================
>   class VCloudConnection(ConnectionUserAndKey):
>   <snip>
>
>       def _get_auth_token(self):
>         ...
>         <snip>
>             body = ET.XML(resp.read())  <---
>         <snip>
>         ...
>   <snip>
>   =================
>
>
> Through various print() and type() statements, I see that "resp.read()", which
> when expanded translates to "http.client.HTTPResponse.read()", returns a
> bytes string. However, I'm wondering if this may also need to be similarly
> ".decode('utf-8')", like this:
>
>      body = ET.XML(resp.read().decode('utf-8'))
>
>
> since the Python-3 documentation for "xml.etree.ElementTree.XML()"
> (which is what ET.XML() expands to), indicates that it is expecting a string
> (not a bytes encoded string).
>
> So I tried the modification above. Here is the before and after:
>
>    resp.read()  [ BEFORE .decode() inserted ]
>
> <class 'bytes'>  <--- types is 'bytes', and the following is the actual bytes string...
>
> b'b\'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\\n  <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\\n</OrgL'
>
>
>   resp.read().decode('utf-8')  [ AFTER .decode() inserted ]
>
>
> <class 'str'>  <--- types is 'str', and the following is the actual string...
>
> b'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\n  <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\n</OrgL
>
>
> Looking carefully at both the bytes string (un-decoded) and the string (decoded) above -- forgetting about whether we
> decode() it or not for the moment -- both sequences appear to be malformed... They both have an extra "b'"
> pre-pending it, and both are truncated at the end (i.e. the enclosing "</OrgL" ... doesn't end correctly). Note: Just
>
> in case this was a print() artefact, I also wrote the output to a file (same truncation issue).
>
> This above issue is only true for python-3. Python 2, (decode()ed or not) emits output correctly as:
>
>    <OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/></OrgList>
>
>
> So there appears to be two problems here:
>    (1) missing ".decode('utf-8')" method call.
>    (2) the output of "resp.read()", with or without  the ".decode('utf-8')" part, appears to be malformed.
>
>
> To make matters more interesting, note that the python-3 case curl(1) output shows a result that is not truncated,
> but still, however, has the pre-pending "b'" (so it's still malformed). And here is that curl(1) output
> (remembering that the above were from print() debug statements of mine, not from curl(1)):
>
> b'<OrgList xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">\n  <Org href="https://services.vcloudexpress.terremark.com/api/v0.8/org/xxxx" type="application/vnd.vmware.vcloud.org+xml" name="email@example.com"/>\n</OrgList>'
>
>
> As I look more for the cause, any helpful feedback is appreciated.
>
>
> Again,... the libcloud API is v0.10.1 with Python v3.2.3
>
>
> Thank you,
> Noel
>
>
>>________________________________
>> From: Sengor <se...@gmail.com>
>>To: users@libcloud.apache.org; Noel Milton Vega <nm...@computingarchitects.com>
>>Sent: Thursday, June 21, 2012 10:21 PM
>>Subject: Re: Terremark authentication example using libcloud ...
>>
>>Hi Noel,
>>
>>Replies in line below:
>>
>>On 22 June 2012 09:30, Noel Milton Vega <nm...@computingarchitects.com> wrote:
>>> (1) I was surprised that that base64 line worked at all (even in Python 2)
>>>     since I always thought use of "b" only worked for string literals
>>>     (not generated strings).
>>
>>Patch has been applied to trunk which now works for both Python 2 & 3.
>>For more info see: https://issues.apache.org/jira/browse/LIBCLOUD-204
>>
>>
>>> (2) Not an issue for this specific thread, but even when I hard-coded
>>>     my auth string (just to get past the Python-3 related auth issue), sadly
>>>     another unrelated (xml parsing) issue arose almost immediately.
>>>     Just as a heads up, it was this:
>>>
>>>
>>> Traceback (most recent call last):
>>>   File "/usr/lib64/python3.2/xml/etree/ElementTree.py", line 1668, in feed
>>>     self._parser.Parse(data, 0)
>>> xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 1
>>>
>>>    (So I never got my list_images() output.)
>>>
>>> (3) Similar to the last point, python-2 gets much further along, but also
>>>     eventually experiences an parse-related exception too (as it's
>>>     iterating through the list the of images too I believe). Again, as a heads-up fyi:
>>>
>>>
>>>   File "/usr/lib/python2.7/site-packages/libcloud/common/base.py", line 76, in __init__
>>>     raise Exception(self.parse_error())
>>> Exception: <Element '{http://www.w3.org/1999/xhtml}html'; at 0x118dd10>
>>
>>Hard to say what the issue could be for these two. Could you post the
>>debugging output for these two (don't forget to remove the private
>>contents such as base64 encoded auth):
>>http://libcloud.apache.org/docs/debugging.html
>>
>>Note I've only used the vCloud module under Python 2.x at the moment
>>and don't have access to Terremark. We did slightly change the VDC
>>object model when vCloud v1.5 support has been introduced. If this is
>>related to what you see it'd be good to fix it up, otherwise it's
>>something else getting in the way.
>>
>>--
>>sengork
>>
>>
>>



-- 
sengork