You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Adam Lamar <ad...@gmail.com> on 2017/02/22 23:42:21 UTC
Expression language and UTC millis
Hi,
I recently noticed some issues with time parsing in the expression language.
On my Linux server configured with UTC time, using UpdateAttribute to
convert a timestamp value to millis works as expected.
Attribute name: timestamp
Sample value: 2017-02-22T23:01:23Z
UpdateAttribute is configured with the following property:
name: timestamp_millis
value: ${timestamp:toDate("yyyy-MM-dd'T'HH:mm:ss'Z'"):toNumber()}
For the above sample value, I get 1487804483000 as timestamp_millis.
epochconverter.com converts timestamp_millis back to the same timestamp
provided above.
On my local OSX dev system with a UTC offset of -07:00, the same timestamp
input yields timestamp_millis as 1487829683000, or the equivalent of
2017-02-23T06:01:23Z.
Is my expression language incorrect, or is something else wrong?
Adam
Re: Expression language and UTC millis
Posted by Andre <an...@fucs.org>.
If I recall correctly, epoch times should always be free from time zone,
always representing seconds/milliseconds since epoch at UTC
So for the original timestamp
2017-02-22T23:01:23Z = 1487804483000 = 2017-02-22T16:01:23+0700
While
2017-02-23T06:01:23Z. =1487829683000 != 2017-02-22T23:01:23Z
There seems to be a 7 hours negative from UTC, suggesting the calculation
occurred on local time?
Seems like a bug to me
On Thu, Feb 23, 2017 at 11:08 AM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:
> Adam,
>
> I think if my math is correct, what you seeing is correct since
> 22T23:01:23 + 7 = 23T06:01:23
>
> Am i missing something?
> Cheers
> Oleg
>
> On Feb 22, 2017, at 6:42 PM, Adam Lamar <ad...@gmail.com> wrote:
>
> Hi,
>
> I recently noticed some issues with time parsing in the expression
> language.
>
> On my Linux server configured with UTC time, using UpdateAttribute to
> convert a timestamp value to millis works as expected.
>
> Attribute name: timestamp
> Sample value: 2017-02-22T23:01:23Z
>
> UpdateAttribute is configured with the following property:
>
> name: timestamp_millis
> value: ${timestamp:toDate("yyyy-MM-dd'T'HH:mm:ss'Z'"):toNumber()}
>
> For the above sample value, I get 1487804483000 as timestamp_millis.
>
> epochconverter.com converts timestamp_millis back to the same timestamp
> provided above.
>
> On my local OSX dev system with a UTC offset of -07:00, the same timestamp
> input yields timestamp_millis as 1487829683000, or the equivalent of
> 2017-02-23T06:01:23Z.
>
> Is my expression language incorrect, or is something else wrong?
>
> Adam
>
>
>
Re: Expression language and UTC millis
Posted by Adam Lamar <ad...@gmail.com>.
Thanks all for your help! Once I built the latest master, specifying 'UTC'
as the second argument of toDate() did the trick:
${timestamp:toDate("yyyy-MM-dd'T'HH:mm:ss'Z'", "UTC"):toNumber()}
Much appreciated,
Adam
Re: Expression language and UTC millis
Posted by Joe Witt <jo...@gmail.com>.
Nice template Andre.
Adam: this will of course only work on the latest master.
Ultimately the SimpleDateFormatter (and even new java.time formatters)
do not appear to actually change the effective timezone of the
formatter object itself. Rather the format string tells it how/what
it will accept as input to parse. If you want it the formatter to be
relative to a specific timezone you have to tell it that explicitly.
I believe Pierre's patch does allow that now and the template seems to
show that. If you do not explicitly set the timezone on the formatter
itself then it will create dates relative to its default system
timezone. So if you dont set it and parse an input value with a
specified zulu time it will still be creating the actual Date with a
timezone offset of your local system. When you go to create the epoch
output then it will have been adjusted by your local TZ and confusion
ensues.
I think Pierre documented this pretty well in his PR but we could
probably provide a series of examples because I do think how several
of us assumed it worked is how it should work.
But, hopefully this gets you going again. Let us know.
Thanks
Joe
On Thu, Feb 23, 2017 at 12:15 AM, Andre <an...@fucs.org> wrote:
> Adam,
>
> Credit goest to JWitt but have you tried:
>
> ${zulu:toDate("yyyy-MM-dd'T'HH:mm:ssXXX", "UTC"):toNumber()}
>
> ?
>
> It is counter intuitive (as you must specify the UTC)
>
> The following template should illustrate
>
> https://gist.github.com/trixpan/f9aaf5081500a6afbcd0e237d9c9c864
>
>
> On Thu, Feb 23, 2017 at 11:59 AM, Andre <an...@fucs.org> wrote:
>>
>> Bryan,
>>
>> Shouldn't we:
>>
>> If TZ contained in string
>> Respect the TZ
>>
>> If no TZ provided:
>>
>> If EL defines TZ:
>> use it
>>
>> Else: Use system
>>
>>
>> ?
>>
>> Wouldn't forcing the user to setup the TZ defeat the purpose of sending
>> dates with TZ already encoded in the string?
>>
>>
>> On Thu, Feb 23, 2017 at 11:49 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> This might be fixed in master by this JIRA, but I haven't tried yet:
>>>
>>> https://issues.apache.org/jira/browse/NIFI-2908
>>>
>>> On Wed, Feb 22, 2017 at 7:39 PM Adam Lamar <ad...@gmail.com> wrote:
>>>>
>>>> Oleg,
>>>>
>>>> A unix epoch timestamp is explicitly defined as the number of seconds
>>>> (or millis) since Jan 1 1970 *UTC*, not any local timezone. Here's an
>>>> example ruby expression from both systems correctly returning the
>>>> 1487804483000 value despite their timezone settings:
>>>>
>>>> irb(main):003:0> Time.parse('2017-02-22T23:01:23Z').to_i*1000
>>>> => 1487804483000
>>>>
>>>> and
>>>>
>>>> 2.1.5 :005 > Time.parse('2017-02-22T23:01:23Z').to_i*1000
>>>> => 1487804483000
>>>>
>>>> The value 1487804483000 is the correct millis for the sample value, so I
>>>> think the question is whether there is a bug in NiFi or whether the
>>>> expression used to parse the date is wrong.
>>>>
>>>> See also:
>>>> http://stackoverflow.com/questions/2853977/unix-timestamp-always-in-gmt
>>>> https://en.wikipedia.org/wiki/Unix_time
>>>>
>>> --
>>> Sent from Gmail Mobile
>>
>>
>
Re: Expression language and UTC millis
Posted by Andre <an...@fucs.org>.
Adam,
Credit goest to JWitt but have you tried:
${zulu:toDate("yyyy-MM-dd'T'HH:mm:ssXXX", "UTC"):toNumber()}
?
It is counter intuitive (as you must specify the UTC)
The following template should illustrate
https://gist.github.com/trixpan/f9aaf5081500a6afbcd0e237d9c9c864
On Thu, Feb 23, 2017 at 11:59 AM, Andre <an...@fucs.org> wrote:
> Bryan,
>
> Shouldn't we:
>
> If TZ contained in string
> Respect the TZ
>
> If no TZ provided:
>
> If EL defines TZ:
> use it
>
> Else: Use system
>
>
> ?
>
> Wouldn't forcing the user to setup the TZ defeat the purpose of sending
> dates with TZ already encoded in the string?
>
>
> On Thu, Feb 23, 2017 at 11:49 AM, Bryan Bende <bb...@gmail.com> wrote:
>
>> This might be fixed in master by this JIRA, but I haven't tried yet:
>>
>> https://issues.apache.org/jira/browse/NIFI-2908
>>
>> On Wed, Feb 22, 2017 at 7:39 PM Adam Lamar <ad...@gmail.com> wrote:
>>
>>> Oleg,
>>>
>>> A unix epoch timestamp is explicitly defined as the number of seconds
>>> (or millis) since Jan 1 1970 *UTC*, not any local timezone. Here's an
>>> example ruby expression from both systems correctly returning the 1487804483000
>>> value despite their timezone settings:
>>>
>>> irb(main):003:0> Time.parse('2017-02-22T23:01:23Z').to_i*1000
>>> => 1487804483000
>>>
>>> and
>>>
>>> 2.1.5 :005 > Time.parse('2017-02-22T23:01:23Z').to_i*1000
>>> => 1487804483000
>>>
>>> The value 1487804483000 is the correct millis for the sample value, so
>>> I think the question is whether there is a bug in NiFi or whether the
>>> expression used to parse the date is wrong.
>>>
>>> See also:
>>> http://stackoverflow.com/questions/2853977/unix-timestamp-always-in-gmt
>>> https://en.wikipedia.org/wiki/Unix_time
>>>
>>> --
>> Sent from Gmail Mobile
>>
>
>
Re: Expression language and UTC millis
Posted by Andre <an...@fucs.org>.
Bryan,
Shouldn't we:
If TZ contained in string
Respect the TZ
If no TZ provided:
If EL defines TZ:
use it
Else: Use system
?
Wouldn't forcing the user to setup the TZ defeat the purpose of sending
dates with TZ already encoded in the string?
On Thu, Feb 23, 2017 at 11:49 AM, Bryan Bende <bb...@gmail.com> wrote:
> This might be fixed in master by this JIRA, but I haven't tried yet:
>
> https://issues.apache.org/jira/browse/NIFI-2908
>
> On Wed, Feb 22, 2017 at 7:39 PM Adam Lamar <ad...@gmail.com> wrote:
>
>> Oleg,
>>
>> A unix epoch timestamp is explicitly defined as the number of seconds (or
>> millis) since Jan 1 1970 *UTC*, not any local timezone. Here's an example
>> ruby expression from both systems correctly returning the 1487804483000
>> value despite their timezone settings:
>>
>> irb(main):003:0> Time.parse('2017-02-22T23:01:23Z').to_i*1000
>> => 1487804483000
>>
>> and
>>
>> 2.1.5 :005 > Time.parse('2017-02-22T23:01:23Z').to_i*1000
>> => 1487804483000
>>
>> The value 1487804483000 is the correct millis for the sample value, so I
>> think the question is whether there is a bug in NiFi or whether the
>> expression used to parse the date is wrong.
>>
>> See also:
>> http://stackoverflow.com/questions/2853977/unix-timestamp-always-in-gmt
>> https://en.wikipedia.org/wiki/Unix_time
>>
>> --
> Sent from Gmail Mobile
>
Re: Expression language and UTC millis
Posted by Bryan Bende <bb...@gmail.com>.
This might be fixed in master by this JIRA, but I haven't tried yet:
https://issues.apache.org/jira/browse/NIFI-2908
On Wed, Feb 22, 2017 at 7:39 PM Adam Lamar <ad...@gmail.com> wrote:
> Oleg,
>
> A unix epoch timestamp is explicitly defined as the number of seconds (or
> millis) since Jan 1 1970 *UTC*, not any local timezone. Here's an example
> ruby expression from both systems correctly returning the 1487804483000
> value despite their timezone settings:
>
> irb(main):003:0> Time.parse('2017-02-22T23:01:23Z').to_i*1000
> => 1487804483000
>
> and
>
> 2.1.5 :005 > Time.parse('2017-02-22T23:01:23Z').to_i*1000
> => 1487804483000
>
> The value 1487804483000 is the correct millis for the sample value, so I
> think the question is whether there is a bug in NiFi or whether the
> expression used to parse the date is wrong.
>
> See also:
> http://stackoverflow.com/questions/2853977/unix-timestamp-always-in-gmt
> https://en.wikipedia.org/wiki/Unix_time
>
> --
Sent from Gmail Mobile
Re: Expression language and UTC millis
Posted by Adam Lamar <ad...@gmail.com>.
Oleg,
A unix epoch timestamp is explicitly defined as the number of seconds (or
millis) since Jan 1 1970 *UTC*, not any local timezone. Here's an example
ruby expression from both systems correctly returning the 1487804483000
value despite their timezone settings:
irb(main):003:0> Time.parse('2017-02-22T23:01:23Z').to_i*1000
=> 1487804483000
and
2.1.5 :005 > Time.parse('2017-02-22T23:01:23Z').to_i*1000
=> 1487804483000
The value 1487804483000 is the correct millis for the sample value, so I
think the question is whether there is a bug in NiFi or whether the
expression used to parse the date is wrong.
See also:
http://stackoverflow.com/questions/2853977/unix-timestamp-always-in-gmt
https://en.wikipedia.org/wiki/Unix_time
Re: Expression language and UTC millis
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Adam,
I think if my math is correct, what you seeing is correct since 22T23:01:23 + 7 = 23T06:01:23
Am i missing something?
Cheers
Oleg
On Feb 22, 2017, at 6:42 PM, Adam Lamar <ad...@gmail.com>> wrote:
Hi,
I recently noticed some issues with time parsing in the expression language.
On my Linux server configured with UTC time, using UpdateAttribute to convert a timestamp value to millis works as expected.
Attribute name: timestamp
Sample value: 2017-02-22T23:01:23Z
UpdateAttribute is configured with the following property:
name: timestamp_millis
value: ${timestamp:toDate("yyyy-MM-dd'T'HH:mm:ss'Z'"):toNumber()}
For the above sample value, I get 1487804483000 as timestamp_millis.
epochconverter.com<http://epochconverter.com/> converts timestamp_millis back to the same timestamp provided above.
On my local OSX dev system with a UTC offset of -07:00, the same timestamp input yields timestamp_millis as 1487829683000, or the equivalent of 2017-02-23T06:01:23Z.
Is my expression language incorrect, or is something else wrong?
Adam