You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by Shri Javadekar <no...@github.com> on 2014/07/26 02:07:26 UTC

[jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

S3 compatible blobStores sometimes return date in the format:
&quot;2014-07-23T20:53:17+0000&quot; instead of the more common
&quot;2014-07-23T18:09:39.944Z&quot;. This caused jclouds to barf with an
IllegalArgumentException.

This commit tries to parse both the formats for S3. The exception
is thrown if both fail.

Added unit tests for the same.
You can merge this Pull Request by running:

  git pull https://github.com/maginatics/jclouds parse-date-seconds-format

Or you can view, comment on it, or merge it online at:

  https://github.com/jclouds/jclouds/pull/456

-- Commit Summary --

  * Try iso8601SecondsDateParse if iso8601DateParse fails.

-- File Changes --

    M apis/s3/src/main/java/org/jclouds/s3/xml/CopyObjectHandler.java (12)
    M apis/s3/src/main/java/org/jclouds/s3/xml/ListAllMyBucketsHandler.java (12)
    M apis/s3/src/main/java/org/jclouds/s3/xml/ListBucketHandler.java (12)
    M apis/s3/src/test/java/org/jclouds/s3/xml/CopyObjectHandlerTest.java (19)
    M apis/s3/src/test/java/org/jclouds/s3/xml/ListBucketHandlerTest.java (20)

-- Patch Links --

https://github.com/jclouds/jclouds/pull/456.patch
https://github.com/jclouds/jclouds/pull/456.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Phillips <no...@github.com>.
> @@ -141,4 +141,18 @@ public final String rfc1123DateFormat() {
>     public final Date rfc1123DateParse(String toParse) {
>        return rfc1123DateFormat.parseDateTime(toParse).toDate();
>     }
> +
> +   @Override
> +   public Date iso8601DateParseWithOptionalTZ(String toParse)
> +         throws IllegalArgumentException {
> +      try {
> +         return iso8601DateParse(toParse);
> +      } catch (IllegalArgumentException orig) {
> +         try {
> +            return iso8601SecondsDateParse(toParse);
> +         } catch (IllegalArgumentException ie) {

[minor] rename `ie` to `ignored`?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15459610

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

Thanks for investigating this!  Let's clean this commit up as suggested above and reference this quirk in a comment.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431948

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Niraj Tolia <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

@andrewgaul Unfortunately, yes. Both the timestamps @shrinandj pointed out were generated as a part of the same list container operation. I would send wire logs but @shrinandj tells me that jclouds evidently barfs with the IllegalArgumentException before we get to that stage.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431559

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1428](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1428/) SUCCESS
This pull request looks good
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50226631

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
Which vendor violates the S3 API?  Instead of working around this in code, perhaps you could add a configuration knob which toggles the date parsing for a specific vendor?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50219377

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Niraj Tolia <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

@andrewgaul Also just as a note, this patch follows the sentiments here - https://github.com/jclouds/jclouds/pull/105#issuecomment-22520277. Config option might work but one would likely turn it on all the time as it is unclear when a provider might generate this (evidently valid) date format. I cannot seem to find any documentation that states the 8601 format timestamps should be in.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431251

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Phillips <no...@github.com>.
> @@ -107,6 +107,15 @@ public void testIso8601DateParseTz() {
>     }
>  
>     @Test
> +   public void testIso8601OptionalTZDateParse() {
> +      Date dsDate = dateService
> +            .iso8601DateParseWithOptionalTZ(testData[0].iso8601SecondsDateString);
> +      Date secondsDate = dateService
> +            .iso8601SecondsDateParse(testData[0].iso8601SecondsDateString);
> +      assertEquals(dsDate, secondsDate);

Shouldn't this rather test that the new method can parse both `testData[0].iso8601DateString` **and** `testData[0].iso8601SecondsDateString` and, that if an invalid string is passed, it throws the *first* exception?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15460021

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Phillips <no...@github.com>.
> @@ -73,6 +73,21 @@
>      */
>     Date iso8601SecondsDateParse(String toParse) throws IllegalArgumentException;
>  
> +   /**
> +    * Parse a given date in either of two iso8601 formats:
> +    * "yyyy-MM-dd'T'HH:mm:ssZ" or "yyyy-MM-dd'T'HH:mm:ss.SSSZ". The latter one
> +    * has the timezone designator, e.g. 2014-07-23T20:53:17+0000. At least one
> +    * S3 compatible blobstore uses both these formats when returning
> +    * container/object metadata.
> +    *
> +    * @param toParse
> +    *           The string to parse.
> +    * @return the Date object of the parsed string.
> +    * @throws IllegalArgumentException
> +    */
> +   Date iso8601DateParseWithOptionalTZ(String toParse)

[minor] Looking at the naming of the others, would this better be `iso8601DateWithOptionalTZParse`?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15459522

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1425](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1425/) ABORTED

[(what's this?)](https://www.cloudbees.com/what-is-buildhive)

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50220435

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
> @@ -201,4 +203,18 @@ public final Date rfc1123DateParse(String toParse) throws IllegalArgumentExcepti
>           }
>        }
>     }
> +
> +   @Override
> +   public Date iso8601DateParseWithOptionalTZ(String toParse)
> +         throws IllegalArgumentException {
> +      try {
> +         return iso8601DateParse(toParse);

Call `iso8601DateParseWithOptionalTZ`?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15434971

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Niraj Tolia <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

@andrewgaul @shrinandj I did some digging and it looks like AWS code itself does alternate parsing - https://github.com/aws/aws-sdk-java/blob/master/src/main/java/com/amazonaws/util/DateUtils.java#L84. I haven't dug into both formats there but it seems to match what we are doing here. 

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431883

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #1033](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/1033/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50219001

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

Could you encapsulate this duplicated logic in a helper method, e.g., `iso8601DateParseWithOptionalSeconds`?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431204

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
Closed #456.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#event-145973428

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
Tested against aws-s3 and pushed to master as d57bbebe7e0217b9a3818c6e1ed460c90469da78.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50256377

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

Does your vendor generate *both* time variants?  Presumably the configuration would enable or or the other parsing.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431274

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1508](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1508/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50255947

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParseWithOptionalTZ(currentOrNull(currentText));

Call `iso8601DateParseWithOptionalTZ`?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15434968

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
@shrinandj I pushed this to 1.7.x as 8ce9f79fde4497d1131f4996c1a0f1006c59d6e8 although I do not believe we plan any further 1.7.x releases.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50292939

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

Actually, a configuration knob seems like a better approach.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431217

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Shri Javadekar <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParseWithOptionalTZ(currentOrNull(currentText));

Done.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15437217

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Phillips <no...@github.com>.
> @@ -201,4 +203,18 @@ public final Date rfc1123DateParse(String toParse) throws IllegalArgumentExcepti
>           }
>        }
>     }
> +
> +   @Override
> +   public Date iso8601DateParseWithOptionalTZ(String toParse)
> +         throws IllegalArgumentException {
> +      try {
> +         return iso8601DateParse(toParse);
> +      } catch (IllegalArgumentException orig) {
> +         try {
> +            return iso8601SecondsDateParse(toParse);
> +         } catch (IllegalArgumentException ie) {

[minor] rename `ie` to `ignored`?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15459602

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #1035](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/1035/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50227130

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1505](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1505/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50219263

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1430](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1430/) SUCCESS
This pull request looks good
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50255686

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Shri Javadekar <no...@github.com>.
Thanks for pushing this to master @andrewgaul. I believe the jclouds 1.8.0 release next week will be based off of master so this fix will make it there. Do we need to backport this fix to the 1.7.x branch for any further 1.7.x releases?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50284492

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Andrew Gaul <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

Please rework this pull request with an `iso8601OptionalMilliSecondsDateParse` helper instead of duplicating exception handling throughout the code.  Also add a comment explaining why this workaround exists and for which vendor and operations.  Have you reported this deviation to the vendor yet?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431651

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by Niraj Tolia <no...@github.com>.
> @@ -51,7 +51,17 @@ public void endElement(String uri, String name, String qName) {
>        if (qName.equals("ETag")) {
>           this.currentETag = currentOrNull(currentText);
>        } else if (qName.equals("LastModified")) {
> -         this.currentLastModified = dateParser.iso8601DateParse(currentOrNull(currentText));
> +         try {
> +            this.currentLastModified = dateParser
> +                  .iso8601DateParse(currentOrNull(currentText));
> +         } catch (IllegalArgumentException orig) {
> +            try {
> +               this.currentLastModified = dateParser
> +                     .iso8601SecondsDateParse(currentOrNull(currentText));

@andrewgaul: Yes, this issue has been raised upstream with EMC but we don't have a resolution yet. If you have any good pointers to documentation for S3 timestamp formats, please do point them my way. I will also post updates in case I hear anything back from them.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456/files#r15431698

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1507](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1507/) UNSTABLE
Looks like there's a problem with this pull request

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50227037

Re: [jclouds] Try iso8601SecondsDateParse if iso8601DateParse fails. (#456)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #1036](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/1036/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/456#issuecomment-50255887