You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/08/20 11:30:00 UTC
[jira] [Commented] (LIBCLOUD-1007) RFC3339 date parsing in GCE
driver fails
[ https://issues.apache.org/jira/browse/LIBCLOUD-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16585814#comment-16585814 ]
ASF GitHub Bot commented on LIBCLOUD-1007:
------------------------------------------
GitHub user fatmcgav opened a pull request:
https://github.com/apache/libcloud/pull/1232
[LIBCLOUD-1007] Resolve RFC3339 date parsing issues with GCE Compute driver.
Resolves issue with RFC3339 date parsing in the GCE Compute driver.
Timestamp strings are now parsed upto and including the microsecond value.
The call to deprecate the Google Compute image now also correctly formats the date as RFC3339 format.
Fixes https://issues.apache.org/jira/browse/LIBCLOUD-1007
- done, ready for review
- [ ] [Code linting](http://libcloud.readthedocs.org/en/latest/development.html#code-style-guide) (required, can be done after the PR checks)
- [ ] Documentation
- [ ] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html)
- [ ] [ICLA](http://libcloud.readthedocs.org/en/latest/development.html#contributing-bigger-changes) (required for bigger changes)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fatmcgav/libcloud gce_fix_rfc3339_parsing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/libcloud/pull/1232.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1232
----
commit d3ae06dcb95725db7d6474f193e182e3e4aa46fc
Author: Gavin Williams <fa...@...>
Date: 2018-08-20T11:20:22Z
[gce] Fix parsing of timestamp string.
Now parses microseconds as part of strptime function, stopping at the '+' character to denote a timezone offset.
commit 37a6d9adccbe0f6b1373708a4f7fcc14a38642b7
Author: Gavin Williams <fa...@...>
Date: 2018-08-20T11:21:31Z
[gce] Use return value from `timestamp_to_datetime` and correctly format
the request datetime to RFC3339 format.
----
> RFC3339 date parsing in GCE driver fails
> ----------------------------------------
>
> Key: LIBCLOUD-1007
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-1007
> Project: Libcloud
> Issue Type: Bug
> Components: Compute
> Reporter: Gavin Williams
> Priority: Major
>
> When attempting to use the `deprecated` param on `GCENodeImage.deprecate()` [1], the call was failing with:
> {noformat}
> 2018-08-20 11:52:08 - DEBUG: Delete at = 2018-08-23T10:52:08.061384+00:00
> Traceback (most recent call last):
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py", line 6371, in ex_deprecate_image
> timestamp_to_datetime(value)
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py", line 57, in timestamp_to_datetime
> ts = datetime.datetime.strptime(timestamp[:-10], '%Y-%m-%dT%H:%M:%S')
> File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py", line 565, in _strptime_datetime
> tt, fraction = _strptime(data_string, format)
> File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py", line 365, in _strptime
> data_string[found.end():])
> ValueError: unconverted data remains: .06
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "scripts/packer-image-manager.py", line 135, in <module>
> main()
> File "scripts/packer-image-manager.py", line 126, in main
> deprecate_images()
> File "scripts/packer-image-manager.py", line 56, in deprecate_images
> dry_run=args.dry_run)
> File "/Users/fatmcgav/Work/infra/infra/packer2.0/lib/utils.py", line 82, in deprecate_image
> if image.deprecate(replacement_image, 'DEPRECATED', deleted=delete_at):
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py", line 699, in deprecate
> deprecated, obsolete, deleted)
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py", line 6375, in ex_deprecate_image
> attribute)
> ValueError: deleted must be an RFC3339 timestamp
> {noformat}
> It would appear that the date parsing logic in `timestamp_to_datetime` [2] is flawed, assuming that the length of `microseconds + tz offset` will always be 10 characters, whereas the above example has 12 characters.
> Patch applied as follows:
> {code:java}
> diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/gce.py
> index 3dfc87d3..efcdcb87 100644
> --- a/libcloud/compute/drivers/gce.py
> +++ b/libcloud/compute/drivers/gce.py
> @@ -52,9 +52,9 @@ def timestamp_to_datetime(timestamp):
> :return: Datetime object corresponding to timestamp
> :rtype: :class:`datetime.datetime`
> """
> - # We remove timezone offset and microseconds (Python 2.5 strptime doesn't
> - # support %f)
> - ts = datetime.datetime.strptime(timestamp[:-10], '%Y-%m-%dT%H:%M:%S')
> + # We remove timezone offset
> + ts = datetime.datetime.strptime(timestamp[:timestamp.index('+')],
> + '%Y-%m-%dT%H:%M:%S.%f')
> tz_hours = int(timestamp[-5:-3])
> tz_mins = int(timestamp[-2:]) * int(timestamp[-6:-5] + '1')
> tz_delta = datetime.timedelta(hours=tz_hours, minutes=tz_mins)
> {code}
>
> However once that issue was resolved, the Google Compute API call failed with:
> {noformat}
> 2018-08-20 12:04:59 - DEBUG: Delete at = 2018-08-23T11:04:59.525758+00:00
> Traceback (most recent call last):
> File "scripts/packer-image-manager.py", line 135, in <module>
> main()
> File "scripts/packer-image-manager.py", line 126, in main
> deprecate_images()
> File "scripts/packer-image-manager.py", line 56, in deprecate_images
> dry_run=args.dry_run)
> File "/Users/fatmcgav/Work/infra/infra/packer2.0/lib/utils.py", line 82, in deprecate_image
> if image.deprecate(replacement_image, 'DEPRECATED', deleted=delete_at):
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py", line 699, in deprecate
> deprecated, obsolete, deleted)
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py", line 6381, in ex_deprecate_image
> self.connection.request(request, method='POST', data=image_data).object
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py", line 124, in request
> response = super(GCEConnection, self).request(*args, **kwargs)
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/common/google.py", line 808, in request
> *args, **kwargs)
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/common/base.py", line 637, in request
> response = responseCls(**kwargs)
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/common/base.py", line 155, in __init__
> self.object = self.parse_body()
> File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/common/google.py", line 288, in parse_body
> raise InvalidRequestError(message, self.status, code)
> libcloud.common.google.InvalidRequestError: {'domain': 'global', 'reason': 'invalid', 'message': "Invalid value for field 'deprecationStatus.deleted': '2018-08-23T11:04:59.525758'. Must be a string representation of a full-date or date-time in valid RFC3339 format with either 0 or 3 digits for fractional seconds"}
> {noformat}
> Tracked this down to the `ex_deprecate_image` [3] call, which appeared to be 'validating' the format of the timestamp, but not actually storing the converted result.
> Another patch applied:
> {code:java}
> diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/gce.py
> index 3dfc87d3..a3329133 100644
> --- a/libcloud/compute/drivers/gce.py
> +++ b/libcloud/compute/drivers/gce.py
> @@ -6523,11 +6523,11 @@ class GCENodeDriver(NodeDriver):
> continue
> try:
> - timestamp_to_datetime(value)
> + value = timestamp_to_datetime(value)
> except:
> raise ValueError('%s must be an RFC3339 timestamp' %
> attribute)
> - image_data[attribute] = value
> + image_data[attribute] = value.isoformat(timespec='milliseconds')
> request = '/global/images/%s/deprecate' % (image.name)
> {code}
> Pull request incoming with both patches.
> [1] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/gce.py#L6485-L6486]
> [2] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/gce.py#L44-L61]
> [3] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/gce.py#L6525-L6530]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)