You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Jason E Bailey <je...@apache.org> on 2018/04/18 16:20:11 UTC
[Discussion] Date handling inconsistencies - resend
I'm re-sending this as I had a problem with my webmail and the formatting got messed up for the text view.
We have inconsistencies on how dates are presented and the formats that are acceptable. This only includes inconsistent handling of the time zone offset. Fixing these would mean a change to the respective bundles to support java 8 by default or to make changes to what the current default format is.
The problem:
Prior to Java 8 there was no out of the box way to support the parsing of a String into a Date that would keep the time zone offset. The java.util.Date object does not contain information about the offset and assumes everything from a UTC. The Calendar object does maintain a time zone but uses the system time zone as a default.
There is a Standard for time strings called ISO-8601 which is used by the JCR, there is a utility class used by several bundles that utilizes this class and this will create a Calendar object with the correct time zone offset. However not all bundles use this.
Current Areas of Concern:
Content Loader with a JSON payload, only recognizes a value as a Date if it uses the ISO-8601 format, however the Content Loader uses the java SimpleDateFormatter instead of the ISO8601 util so time zone offset it lost.
Content Parser recognizes ISO-8601 and an ECMA format, ECMA format does not support offset.
Form Post handling with a TypeHint of Date will recognize multiple formats including ISO-8601, will only keep the time zone offset with the ISO-8601 format.
Default JSON handler is returning dates as an ECMA format, not as ISO-8601 the ECMA format is a format that was primarily used by browsers. This format is not the default for browsers anymore, the default format for browsers is the ISO-8601 format.
Solutions:
The easiest way of parsing date strings and keeping offsets would be to use the java.time package from java 8 which would upgrade the dependencies in those bundles mentioned.
Other options would be to have any usage of the ISO-8601 format be parsed with the ISO8601 utility class and to have ALL exports of Date Strings use the 8601 format as default. We could then state that if you want to keep the offset use the standard date string format. That would require a change in the default JSON handler.
Other options and opinions would be great.
- Jason
Re: [Discussion] Date handling inconsistencies - resend
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Sun, May 6, 2018 at 7:21 PM, Jason E Bailey <je...@apache.org> wrote:
> ...I think the appropriate choice is to have it default to ISO-8601 formatting
> while providing an option to "enable support for the deprecated
> ECMA formatting"...
+1 assuming tests cover both options, thanks for investigating!
-Bertrand
RE: [Discussion] Date handling inconsistencies - resend
Posted by Stefan Seifert <ss...@pro-vision.de>.
hello jason.
yes, this makes sense.
stefan
>-----Original Message-----
>From: Jason E Bailey [mailto:jeb@apache.org]
>Sent: Sunday, May 6, 2018 7:21 PM
>To: dev@sling.apache.org
>Subject: Re: [Discussion] Date handling inconsistencies - resend
>
>I've been giving it a lot of thought and I think the appropriate choice is
>to have it default to ISO-8601 formatting while providing an option to
>"enable support for the deprecated ECMA formatting"
>
>If we don't use ISO-8601 by default we're just delaying issues until we do.
>I think it's better to just go ahead and do it while still allowing people
>the option to get the older format while stating that support in a way to
>imply it will go away at some point.
>
>- Jason
>
>On Thu, May 3, 2018, at 3:35 AM, Stefan Seifert wrote:
>> maybe this would be a sensible first step - although it would be fine to
>> have it by default to ISO-8601 in the future for new applications.
>>
>> stefan
>>
>> >-----Original Message-----
>> >From: Jason E Bailey [mailto:jeb@apache.org]
>> >Sent: Monday, April 23, 2018 8:37 PM
>> >To: dev@sling.apache.org
>> >Subject: Re: [Discussion] Date handling inconsistencies - resend
>> >
>> >For the JSON rendering, would a service configuration to enable ISO8601
>> >date support be sufficient initially?
>> >
>> >
>> >- Jason
>> >
>> >On Mon, Apr 23, 2018, at 12:58 PM, Jason E Bailey wrote:
>> >> I'll just come out and say that someone, somewhere, will have
>something
>> >> break because of this. Because they wrote something that is very
>> >> specific to a particular use case.
>> >>
>> >> From a loading of content perspective, the two changes I made keep the
>> >> same instance in time, they just correctly store the offset that was
>> >> provided. If someone has written a test or some code that manipulated
>> >> that date based on the belief that no offset was recorded, that may be
>> >> impacted. I feel this is an edge case enough that I'm good with the
>> >> changes.
>> >>
>> >> There are two areas that are sensitive enough that I'm not going to
>> >> directly commit without conversation.
>> >>
>> >> One is POST handler, where the only format that handles the offset
>> >> correctly is the ISO-8601 format. Although I don't expect the content
>> >> loaders to be problematic, this fix would be larger then what I can
>get
>> >> to for a while.
>> >>
>> >> The other is the default JSON rendering for the GET request. There is
>no
>> >> specification in our documents and no tests that check to see how the
>> >> Date string is supposed to be generated. There is no JSON
>specification
>> >> for a Date. The XML version of the data is using ISO-8601. So my
>> >> assumption on the original development of this is that the JSON was
>> >> being generated in a way that would be easily consumable by browsers
>at
>> >> the time. However the format used is now deprecated. Everyone is
>moving
>> >> to the ISO-8601 format or, at the very least, their interpretation of
>> >> the spec.
>> >>
>> >> That one is potentially the biggest problem, because it's the one
>that's
>> >> most likely to break something. Potential issues would be something
>> >> like downstream UI components. It's hard for me to say and I would
>have
>> >> to do some testing.
>> >>
>> >>
>> >> - Jason
>> >>
>> >
>>
Re: [Discussion] Date handling inconsistencies - resend
Posted by Jason E Bailey <je...@apache.org>.
I've been giving it a lot of thought and I think the appropriate choice is to have it default to ISO-8601 formatting while providing an option to "enable support for the deprecated ECMA formatting"
If we don't use ISO-8601 by default we're just delaying issues until we do. I think it's better to just go ahead and do it while still allowing people the option to get the older format while stating that support in a way to imply it will go away at some point.
- Jason
On Thu, May 3, 2018, at 3:35 AM, Stefan Seifert wrote:
> maybe this would be a sensible first step - although it would be fine to
> have it by default to ISO-8601 in the future for new applications.
>
> stefan
>
> >-----Original Message-----
> >From: Jason E Bailey [mailto:jeb@apache.org]
> >Sent: Monday, April 23, 2018 8:37 PM
> >To: dev@sling.apache.org
> >Subject: Re: [Discussion] Date handling inconsistencies - resend
> >
> >For the JSON rendering, would a service configuration to enable ISO8601
> >date support be sufficient initially?
> >
> >
> >- Jason
> >
> >On Mon, Apr 23, 2018, at 12:58 PM, Jason E Bailey wrote:
> >> I'll just come out and say that someone, somewhere, will have something
> >> break because of this. Because they wrote something that is very
> >> specific to a particular use case.
> >>
> >> From a loading of content perspective, the two changes I made keep the
> >> same instance in time, they just correctly store the offset that was
> >> provided. If someone has written a test or some code that manipulated
> >> that date based on the belief that no offset was recorded, that may be
> >> impacted. I feel this is an edge case enough that I'm good with the
> >> changes.
> >>
> >> There are two areas that are sensitive enough that I'm not going to
> >> directly commit without conversation.
> >>
> >> One is POST handler, where the only format that handles the offset
> >> correctly is the ISO-8601 format. Although I don't expect the content
> >> loaders to be problematic, this fix would be larger then what I can get
> >> to for a while.
> >>
> >> The other is the default JSON rendering for the GET request. There is no
> >> specification in our documents and no tests that check to see how the
> >> Date string is supposed to be generated. There is no JSON specification
> >> for a Date. The XML version of the data is using ISO-8601. So my
> >> assumption on the original development of this is that the JSON was
> >> being generated in a way that would be easily consumable by browsers at
> >> the time. However the format used is now deprecated. Everyone is moving
> >> to the ISO-8601 format or, at the very least, their interpretation of
> >> the spec.
> >>
> >> That one is potentially the biggest problem, because it's the one that's
> >> most likely to break something. Potential issues would be something
> >> like downstream UI components. It's hard for me to say and I would have
> >> to do some testing.
> >>
> >>
> >> - Jason
> >>
> >
>
RE: [Discussion] Date handling inconsistencies - resend
Posted by Stefan Seifert <ss...@pro-vision.de>.
maybe this would be a sensible first step - although it would be fine to have it by default to ISO-8601 in the future for new applications.
stefan
>-----Original Message-----
>From: Jason E Bailey [mailto:jeb@apache.org]
>Sent: Monday, April 23, 2018 8:37 PM
>To: dev@sling.apache.org
>Subject: Re: [Discussion] Date handling inconsistencies - resend
>
>For the JSON rendering, would a service configuration to enable ISO8601
>date support be sufficient initially?
>
>
>- Jason
>
>On Mon, Apr 23, 2018, at 12:58 PM, Jason E Bailey wrote:
>> I'll just come out and say that someone, somewhere, will have something
>> break because of this. Because they wrote something that is very
>> specific to a particular use case.
>>
>> From a loading of content perspective, the two changes I made keep the
>> same instance in time, they just correctly store the offset that was
>> provided. If someone has written a test or some code that manipulated
>> that date based on the belief that no offset was recorded, that may be
>> impacted. I feel this is an edge case enough that I'm good with the
>> changes.
>>
>> There are two areas that are sensitive enough that I'm not going to
>> directly commit without conversation.
>>
>> One is POST handler, where the only format that handles the offset
>> correctly is the ISO-8601 format. Although I don't expect the content
>> loaders to be problematic, this fix would be larger then what I can get
>> to for a while.
>>
>> The other is the default JSON rendering for the GET request. There is no
>> specification in our documents and no tests that check to see how the
>> Date string is supposed to be generated. There is no JSON specification
>> for a Date. The XML version of the data is using ISO-8601. So my
>> assumption on the original development of this is that the JSON was
>> being generated in a way that would be easily consumable by browsers at
>> the time. However the format used is now deprecated. Everyone is moving
>> to the ISO-8601 format or, at the very least, their interpretation of
>> the spec.
>>
>> That one is potentially the biggest problem, because it's the one that's
>> most likely to break something. Potential issues would be something
>> like downstream UI components. It's hard for me to say and I would have
>> to do some testing.
>>
>>
>> - Jason
>>
>
Re: [Discussion] Date handling inconsistencies - resend
Posted by Jason E Bailey <je...@apache.org>.
For the JSON rendering, would a service configuration to enable ISO8601 date support be sufficient initially?
- Jason
On Mon, Apr 23, 2018, at 12:58 PM, Jason E Bailey wrote:
> I'll just come out and say that someone, somewhere, will have something
> break because of this. Because they wrote something that is very
> specific to a particular use case.
>
> From a loading of content perspective, the two changes I made keep the
> same instance in time, they just correctly store the offset that was
> provided. If someone has written a test or some code that manipulated
> that date based on the belief that no offset was recorded, that may be
> impacted. I feel this is an edge case enough that I'm good with the
> changes.
>
> There are two areas that are sensitive enough that I'm not going to
> directly commit without conversation.
>
> One is POST handler, where the only format that handles the offset
> correctly is the ISO-8601 format. Although I don't expect the content
> loaders to be problematic, this fix would be larger then what I can get
> to for a while.
>
> The other is the default JSON rendering for the GET request. There is no
> specification in our documents and no tests that check to see how the
> Date string is supposed to be generated. There is no JSON specification
> for a Date. The XML version of the data is using ISO-8601. So my
> assumption on the original development of this is that the JSON was
> being generated in a way that would be easily consumable by browsers at
> the time. However the format used is now deprecated. Everyone is moving
> to the ISO-8601 format or, at the very least, their interpretation of
> the spec.
>
> That one is potentially the biggest problem, because it's the one that's
> most likely to break something. Potential issues would be something
> like downstream UI components. It's hard for me to say and I would have
> to do some testing.
>
>
> - Jason
>
Re: [Discussion] Date handling inconsistencies - resend
Posted by Jason E Bailey <je...@apache.org>.
I see what you're saying. I will get some unit tests in specifically around formatting, which is where the problems might crop up.
- Jason
On Tue, Apr 24, 2018, at 4:24 AM, Bertrand Delacretaz wrote:
> Hi Jason,
>
> On Mon, Apr 23, 2018 at 6:58 PM, Jason E Bailey <je...@apache.org> wrote:
> > ...I'll just come out and say that someone, somewhere, will have something break because of this...
>
> Could you write some simple tests that demonstrate those potential breakages?
>
> I think that's the most precise way of expressing those things -
> either tests on the current code, that your changes will break and
> require slightly changing, or tests that demonstrate the edge cases.
>
> -Bertrand
Re: [Discussion] Date handling inconsistencies - resend
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi Jason,
On Mon, Apr 23, 2018 at 6:58 PM, Jason E Bailey <je...@apache.org> wrote:
> ...I'll just come out and say that someone, somewhere, will have something break because of this...
Could you write some simple tests that demonstrate those potential breakages?
I think that's the most precise way of expressing those things -
either tests on the current code, that your changes will break and
require slightly changing, or tests that demonstrate the edge cases.
-Bertrand
Re: [Discussion] Date handling inconsistencies - resend
Posted by Jason E Bailey <je...@apache.org>.
I'll just come out and say that someone, somewhere, will have something break because of this. Because they wrote something that is very specific to a particular use case.
From a loading of content perspective, the two changes I made keep the same instance in time, they just correctly store the offset that was provided. If someone has written a test or some code that manipulated that date based on the belief that no offset was recorded, that may be impacted. I feel this is an edge case enough that I'm good with the changes.
There are two areas that are sensitive enough that I'm not going to directly commit without conversation.
One is POST handler, where the only format that handles the offset correctly is the ISO-8601 format. Although I don't expect the content loaders to be problematic, this fix would be larger then what I can get to for a while.
The other is the default JSON rendering for the GET request. There is no specification in our documents and no tests that check to see how the Date string is supposed to be generated. There is no JSON specification for a Date. The XML version of the data is using ISO-8601. So my assumption on the original development of this is that the JSON was being generated in a way that would be easily consumable by browsers at the time. However the format used is now deprecated. Everyone is moving to the ISO-8601 format or, at the very least, their interpretation of the spec.
That one is potentially the biggest problem, because it's the one that's most likely to break something. Potential issues would be something like downstream UI components. It's hard for me to say and I would have to do some testing.
- Jason
On Mon, Apr 23, 2018, at 3:06 AM, Bertrand Delacretaz wrote:
> Hi,
>
> On Fri, Apr 20, 2018 at 3:07 PM, Jason E Bailey <je...@apache.org> wrote:
> > ...I cam to the same conclusion about the Content Loader, I'll make the bug and fix for that either today or
> > tomorrow. For the second point, I will go ahead and make the bug fix, and do a pull request so that
> > we can get additional input on it....
>
> I haven't looked at the details so far - can you do that while staying
> fully backwards compatible, and do we have tests that demonstrate
> that?
>
> I'm asking because incompatibilities in this area could lead to hard
> to troubleshoot issues.
>
> -Bertrand
Re: [Discussion] Date handling inconsistencies - resend
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Fri, Apr 20, 2018 at 3:07 PM, Jason E Bailey <je...@apache.org> wrote:
> ...I cam to the same conclusion about the Content Loader, I'll make the bug and fix for that either today or
> tomorrow. For the second point, I will go ahead and make the bug fix, and do a pull request so that
> we can get additional input on it....
I haven't looked at the details so far - can you do that while staying
fully backwards compatible, and do we have tests that demonstrate
that?
I'm asking because incompatibilities in this area could lead to hard
to troubleshoot issues.
-Bertrand
Re: [Discussion] Date handling inconsistencies - resend
Posted by Jason E Bailey <je...@apache.org>.
I cam to the same conclusion about the Content Loader, I'll make the bug and fix for that either today or tomorrow. For the second point, I will go ahead and make the bug fix, and do a pull request so that we can get additional input on it. But I believe it's one of those things that will need to be done at some point and I'd prefer sooner then later.
I looked up the ECMA specification and the specification for the Date format is IS0-8601 so if someone is relying on the JSON response to be handled correctly at the browser level, what we are supplying is currently supported only by the grace of the browser manufacturers.
- Jason
On Fri, Apr 20, 2018, at 4:05 AM, Stefan Seifert wrote:
> hello jason.
>
> thanks for this comprehensive summary.
>
> in my opinion for the long run the best solution is to use ISO-8601 for
> all locations and deprecated the ECMA format and discourage usage of it
> (but keep it where it's parsed for backward compatibility).
>
> in this way we have two open points:
>
> 1. Content Loader supports ISO-8601, but does not correctly handle the
> timezones. this can be fixed by using the jackrabbit util class used in
> the other places already. we should create a bug ticket and fix it right
> away.
>
> 2. the only real problematic place is the GET servlet/Default JSON
> handler which currently outputs dates in ECMA by default. we could
> change this easily, but this will break backwards compatibility of code
> relying on it's output. in a very central piece of code.
>
> once we are on java 8 (you opened a separate thread for this) we can fix
> the ECMA parsing support easily. and alternative for java 7 would
> probably be to use a thirdparty library like joda-time (have not checked
> if this really solves this) - but this would mean an additionally
> dependency in the starter/launchpad.
>
> stefan
>
>
RE: [Discussion] Date handling inconsistencies - resend
Posted by Stefan Seifert <ss...@pro-vision.de>.
hello jason.
thanks for this comprehensive summary.
in my opinion for the long run the best solution is to use ISO-8601 for all locations and deprecated the ECMA format and discourage usage of it (but keep it where it's parsed for backward compatibility).
in this way we have two open points:
1. Content Loader supports ISO-8601, but does not correctly handle the timezones. this can be fixed by using the jackrabbit util class used in the other places already. we should create a bug ticket and fix it right away.
2. the only real problematic place is the GET servlet/Default JSON handler which currently outputs dates in ECMA by default. we could change this easily, but this will break backwards compatibility of code relying on it's output. in a very central piece of code.
once we are on java 8 (you opened a separate thread for this) we can fix the ECMA parsing support easily. and alternative for java 7 would probably be to use a thirdparty library like joda-time (have not checked if this really solves this) - but this would mean an additionally dependency in the starter/launchpad.
stefan
>-----Original Message-----
>From: Jason E Bailey [mailto:jeb@apache.org]
>Sent: Wednesday, April 18, 2018 6:20 PM
>To: dev@sling.apache.org
>Subject: [Discussion] Date handling inconsistencies - resend
>
>I'm re-sending this as I had a problem with my webmail and the formatting
>got messed up for the text view.
>
>We have inconsistencies on how dates are presented and the formats that are
>acceptable. This only includes inconsistent handling of the time zone
>offset. Fixing these would mean a change to the respective bundles to
>support java 8 by default or to make changes to what the current default
>format is.
>
>The problem:
>Prior to Java 8 there was no out of the box way to support the parsing of a
>String into a Date that would keep the time zone offset. The java.util.Date
>object does not contain information about the offset and assumes everything
>from a UTC. The Calendar object does maintain a time zone but uses the
>system time zone as a default.
>
>There is a Standard for time strings called ISO-8601 which is used by the
>JCR, there is a utility class used by several bundles that utilizes this
>class and this will create a Calendar object with the correct time zone
>offset. However not all bundles use this.
>
>Current Areas of Concern:
>Content Loader with a JSON payload, only recognizes a value as a Date if
>it uses the ISO-8601 format, however the Content Loader uses the java
>SimpleDateFormatter instead of the ISO8601 util so time zone offset it
>lost.
>
>Content Parser recognizes ISO-8601 and an ECMA format, ECMA format does not
>support offset.
>
>Form Post handling with a TypeHint of Date will recognize multiple formats
>including ISO-8601, will only keep the time zone offset with the ISO-8601
>format.
>
>Default JSON handler is returning dates as an ECMA format, not as ISO-8601
>the ECMA format is a format that was primarily used by browsers. This
>format is not the default for browsers anymore, the default format for
>browsers is the ISO-8601 format.
>
>Solutions:
>The easiest way of parsing date strings and keeping offsets would be to use
>the java.time package from java 8 which would upgrade the dependencies in
>those bundles mentioned.
>
>Other options would be to have any usage of the ISO-8601 format be parsed
>with the ISO8601 utility class and to have ALL exports of Date Strings use
>the 8601 format as default. We could then state that if you want to keep
>the offset use the standard date string format. That would require a change
>in the default JSON handler.
>
>Other options and opinions would be great.
>
>- Jason