You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Markus Weimer <ma...@weimo.de> on 2017/07/25 15:23:30 UTC
[Discuss] Harmonized time stamp format
Hi,
right now, we don't use the same timestamp format in our logs across
Java and .NET, and we aren't even consistent within one language
environment. I believe we should strive to harmonize that, but am not
quite sure what format to settle on.
How about this as a strawman: YYYY-MM-DD-HH-SS-MMM, e.g. 2017-07-25-08-23-123
Would this work? If not, what should be our standard?
Thanks,
Markus
Re: [Discuss] Harmonized time stamp format
Posted by Rogan Carr <ro...@gmail.com>.
Correction:
.NET format is
const string dotNetIso8601Format = "yyyy-MM-ddTHH:mm:ss.fffK";
The K specifier maintains UTC formats.
Best,
Rogan
On Tue, Jul 25, 2017 at 9:56 AM, Rogan Carr <ro...@gmail.com> wrote:
> Hi All,
>
> The inconsistency in time formats has been something that's bothered me
> about parsing REEF logs, and this is something that I'd love to have
> standardized.
>
> I've worked with real-time streaming applications in the past, and the
> requirements we usually settle on are:
>
> * Needs to be supported across platforms and languages
>
> * Needs to be human readable.
> If you have to copy/paste/parse to determine the time of event, it's
> next to worthless for debugging.
>
> * Needs to contain the timezone offset
> Technically, servers can all use UTC, but if we ever want to ask "what
> time of the day was it when this event happened", we need the offset.
> Especially useful to correlate time-of-day events across different
> timezones. (i.e. Maybe there's some administrative event at 3 am local time
> in different datacenters that affects performance; this won't be detectable
> unless you log the local time.)
>
> * Bonus if it's an international standard format
> Almost guarantees wide library support and recognition, plus people have
> already thought through the details.
>
> I propose using this format: "2017-07-25T09:31:22.365-07:00".
>
> This is an implementation of the ISO8601 format, and an abridged version
> of .NET's "Round-trip Format" [1], where you only report down to the
> millisecond. This timestamp has served me well in the past, and I've never
> run into an instance where it's let me down.
>
> Here are common language format strings for these:
> const string dotNetIso8601Format = "yyyy-MM-ddTHH:mm:ss.fffzzz";
> static final String javaIso8601Format = "yyyy-MM-dd'T'HH:mm:ss.SSSZ";
> const javaScriptIso8601Format = 'YYYY-MM-DDTHH:mm:ss.SSSZZ';
>
> Java doesn't have great timestamp support. This format used
> SimpleDateFormat and required one correction step to insert a colon in the
> timezone offset. I can provide the code if we want to go forward with this.
>
> There is a similar case for Python. I have python format and the necessary
> correction code as well.
>
> Best,
> Rogan
>
> [1] More about the .NET Round-trip Format: https://docs.microsoft.com/en-
> us/dotnet/standard/base-types/how-to-round-trip-date-and-time-values
>
>
> On Tue, Jul 25, 2017 at 8:44 AM, Saikat Kanjilal <sx...@gmail.com>
> wrote:
>
>> Markus,
>> Before rendering feedback on this I wanted to understand the use case
>> some more, are these logs only used by the engineers and customers of reef
>> ? Also if that is the case it seems like a simpler version of a time
>> stamp would suffice as I think this is mainly used for troubleshooting,
>> correct?
>>
>> What usually works for me a simple YYYY-MM-DD-HH-MM-SS when
>> troubleshooting and debugging issues.
>>
>> My 2 pennies.
>>
>>
>>
>> Sent from my iPhone
>>
>> > On Jul 25, 2017, at 8:23 AM, Markus Weimer <ma...@weimo.de> wrote:
>> >
>> > Hi,
>> >
>> > right now, we don't use the same timestamp format in our logs across
>> > Java and .NET, and we aren't even consistent within one language
>> > environment. I believe we should strive to harmonize that, but am not
>> > quite sure what format to settle on.
>> >
>> > How about this as a strawman: YYYY-MM-DD-HH-SS-MMM, e.g.
>> 2017-07-25-08-23-123
>> >
>> > Would this work? If not, what should be our standard?
>> >
>> > Thanks,
>> >
>> > Markus
>>
>
>
Re: [Discuss] Harmonized time stamp format
Posted by Rogan Carr <ro...@gmail.com>.
Hi All,
The inconsistency in time formats has been something that's bothered me
about parsing REEF logs, and this is something that I'd love to have
standardized.
I've worked with real-time streaming applications in the past, and the
requirements we usually settle on are:
* Needs to be supported across platforms and languages
* Needs to be human readable.
If you have to copy/paste/parse to determine the time of event, it's next
to worthless for debugging.
* Needs to contain the timezone offset
Technically, servers can all use UTC, but if we ever want to ask "what
time of the day was it when this event happened", we need the offset.
Especially useful to correlate time-of-day events across different
timezones. (i.e. Maybe there's some administrative event at 3 am local time
in different datacenters that affects performance; this won't be detectable
unless you log the local time.)
* Bonus if it's an international standard format
Almost guarantees wide library support and recognition, plus people have
already thought through the details.
I propose using this format: "2017-07-25T09:31:22.365-07:00".
This is an implementation of the ISO8601 format, and an abridged version of
.NET's "Round-trip Format" [1], where you only report down to the
millisecond. This timestamp has served me well in the past, and I've never
run into an instance where it's let me down.
Here are common language format strings for these:
const string dotNetIso8601Format = "yyyy-MM-ddTHH:mm:ss.fffzzz";
static final String javaIso8601Format = "yyyy-MM-dd'T'HH:mm:ss.SSSZ";
const javaScriptIso8601Format = 'YYYY-MM-DDTHH:mm:ss.SSSZZ';
Java doesn't have great timestamp support. This format used
SimpleDateFormat and required one correction step to insert a colon in the
timezone offset. I can provide the code if we want to go forward with this.
There is a similar case for Python. I have python format and the necessary
correction code as well.
Best,
Rogan
[1] More about the .NET Round-trip Format:
https://docs.microsoft.com/en-us/dotnet/standard/base-types/how-to-round-trip-date-and-time-values
On Tue, Jul 25, 2017 at 8:44 AM, Saikat Kanjilal <sx...@gmail.com> wrote:
> Markus,
> Before rendering feedback on this I wanted to understand the use case some
> more, are these logs only used by the engineers and customers of reef ?
> Also if that is the case it seems like a simpler version of a time stamp
> would suffice as I think this is mainly used for troubleshooting, correct?
>
> What usually works for me a simple YYYY-MM-DD-HH-MM-SS when
> troubleshooting and debugging issues.
>
> My 2 pennies.
>
>
>
> Sent from my iPhone
>
> > On Jul 25, 2017, at 8:23 AM, Markus Weimer <ma...@weimo.de> wrote:
> >
> > Hi,
> >
> > right now, we don't use the same timestamp format in our logs across
> > Java and .NET, and we aren't even consistent within one language
> > environment. I believe we should strive to harmonize that, but am not
> > quite sure what format to settle on.
> >
> > How about this as a strawman: YYYY-MM-DD-HH-SS-MMM, e.g.
> 2017-07-25-08-23-123
> >
> > Would this work? If not, what should be our standard?
> >
> > Thanks,
> >
> > Markus
>
Re: [Discuss] Harmonized time stamp format
Posted by Saikat Kanjilal <sx...@gmail.com>.
Markus,
Before rendering feedback on this I wanted to understand the use case some more, are these logs only used by the engineers and customers of reef ? Also if that is the case it seems like a simpler version of a time stamp would suffice as I think this is mainly used for troubleshooting, correct?
What usually works for me a simple YYYY-MM-DD-HH-MM-SS when troubleshooting and debugging issues.
My 2 pennies.
Sent from my iPhone
> On Jul 25, 2017, at 8:23 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> Hi,
>
> right now, we don't use the same timestamp format in our logs across
> Java and .NET, and we aren't even consistent within one language
> environment. I believe we should strive to harmonize that, but am not
> quite sure what format to settle on.
>
> How about this as a strawman: YYYY-MM-DD-HH-SS-MMM, e.g. 2017-07-25-08-23-123
>
> Would this work? If not, what should be our standard?
>
> Thanks,
>
> Markus