You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Chowdhury, Rifat" <Ri...@disney.com> on 2020/01/23 23:06:34 UTC
Nifi Expression Language Question
Hi,
I have a use case where I am storing a variable called “origin_timezone”. I am trying to convert a String which is originally in EST timezone to UTC time Zone. In my update Attribute, I write the following variable:
"America/New_York"
UTCDateString = ${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", ${origin_timezone}):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
does not properly convert the String
UTCDateString =${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", "America/New_York"):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
while when I hardcode it to “America/New_York” it works perfectly. My use case is that I want to dynamically update UTCDateString so that I may convert any timeZone to UTC not just EST? How may I accomplish this?
Example of what I am trying to do…
Correct response:
UTCDateString
20200122-200000
filename
wdgespcom/06-wdgespcom_20200122-150000.tsv.gz
Incorrect Response:
UTCDateString
20200122-020000
filename
wdgespcom/04-wdgespcom_20200122-020000.tsv.gz
Rifat Chowdhury
Software Engineer, Data Platforms
[signature_1999449656]
THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>
Re: Nifi Expression Language Question
Posted by "Chowdhury, Rifat" <Ri...@disney.com>.
Hi Mark,
I am using Nifi 1.10 on an EKS (Kubernetes) cluster. I read in a lookup csv file store that csv file in Redis. That lookup file contains the names of suites, different suites are from different time zones. For example, suite A is from “America/New_York” and suite B is from “America/Los_Angeles”. Once I read the lookup data from RedisCache, and extract the time zone from the lookup data at that point I would store that timezone in a variable called “origin_timezone”.
For example: if the “origin_timezone” = “America/Los_Angeles”, and the filename = "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" Then my UTCDateString = 20200122-230000<tel:20200122-200000>.
If the “origin_timezone” = “America/New_York”, and the filename = "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" Then my UTCDateString = 20200122-200000<tel:20200122-200000>. But in my Nifi Template, using the “origin_timezone” variable does not properly extract the correct UTCDateString. I have attached my Nifi Template for reference.
Rifat Chowdhury
Software Engineer, Data Platforms
[signature_1999449656]
THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>
From: Mark Payne <ma...@hotmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, January 23, 2020 at 6:30 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Cc: "Rudra, Anshuman X. -ND" <An...@espn.com>, "Loyot, Brendan" <Br...@espn.com>, "Nagwani, Dheeraj X. -ND" <dh...@disney.com>
Subject: Re: Nifi Expression Language Question
Rifat,
I tried the example here. I got the same response whether using "America/New_York" hardcoded or using ${origin_timezone}. When I used the filename "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" I got a value of 20200122-200000<tel:20200122-200000>.
When I used a filename of "wdgespcom/04-wdgespcom_20200122-020000.tsv.gz" I got a value of 20200122-070000, which I believe is correct.
Perhaps I misunderstood what you're trying to do? What version of NiFi are you using?
Thanks
-Mark
On Jan 23, 2020, at 6:06 PM, Chowdhury, Rifat <Ri...@disney.com>> wrote:
Hi,
I have a use case where I am storing a variable called “origin_timezone”. I am trying to convert a String which is originally in EST timezone to UTC time Zone. In my update Attribute, I write the following variable:
"America/New_York"
UTCDateString = ${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", ${origin_timezone}):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
does not properly convert the String
UTCDateString =${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", "America/New_York"):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
while when I hardcode it to “America/New_York” it works perfectly. My use case is that I want to dynamically update UTCDateString so that I may convert any timeZone to UTC not just EST? How may I accomplish this?
Example of what I am trying to do…
Correct response:
UTCDateString
20200122-200000<tel:20200122-200000>
filename
wdgespcom/06-wdgespcom_20200122-150000.tsv.gz
Incorrect Response:
UTCDateString
20200122-020000<tel:20200122-020000>
filename
wdgespcom/04-wdgespcom_20200122-020000.tsv.gz
Rifat Chowdhury
Software Engineer, Data Platforms
<image001.png>
THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>
Re: Nifi Expression Language Question
Posted by "Chowdhury, Rifat" <Ri...@disney.com>.
Never Mind. The mistake I made was When I extracted the string, I also extracted the quotation marks…the Nifi ToDate function was parsing “"America/New_York"” instead of "America/New_York"… that’s why it worked when I hardcoded it 😊 Thanks Mark and Anshuman for your help. Silly on my part.
Rifat Chowdhury
Software Engineer, Data Platforms
[signature_1999449656]
THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>
From: Mark Payne <ma...@hotmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, January 23, 2020 at 6:30 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Cc: "Rudra, Anshuman X. -ND" <An...@espn.com>, "Loyot, Brendan" <Br...@espn.com>, "Nagwani, Dheeraj X. -ND" <dh...@disney.com>
Subject: Re: Nifi Expression Language Question
Rifat,
I tried the example here. I got the same response whether using "America/New_York" hardcoded or using ${origin_timezone}. When I used the filename "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" I got a value of 20200122-200000<tel:20200122-200000>.
When I used a filename of "wdgespcom/04-wdgespcom_20200122-020000.tsv.gz" I got a value of 20200122-070000, which I believe is correct.
Perhaps I misunderstood what you're trying to do? What version of NiFi are you using?
Thanks
-Mark
On Jan 23, 2020, at 6:06 PM, Chowdhury, Rifat <Ri...@disney.com>> wrote:
Hi,
I have a use case where I am storing a variable called “origin_timezone”. I am trying to convert a String which is originally in EST timezone to UTC time Zone. In my update Attribute, I write the following variable:
"America/New_York"
UTCDateString = ${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", ${origin_timezone}):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
does not properly convert the String
UTCDateString =${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", "America/New_York"):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
while when I hardcode it to “America/New_York” it works perfectly. My use case is that I want to dynamically update UTCDateString so that I may convert any timeZone to UTC not just EST? How may I accomplish this?
Example of what I am trying to do…
Correct response:
UTCDateString
20200122-200000<tel:20200122-200000>
filename
wdgespcom/06-wdgespcom_20200122-150000.tsv.gz
Incorrect Response:
UTCDateString
20200122-020000<tel:20200122-020000>
filename
wdgespcom/04-wdgespcom_20200122-020000.tsv.gz
Rifat Chowdhury
Software Engineer, Data Platforms
<image001.png>
THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>
Re: Nifi Expression Language Question
Posted by Mark Payne <ma...@hotmail.com>.
Rifat,
I tried the example here. I got the same response whether using "America/New_York" hardcoded or using ${origin_timezone}. When I used the filename "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" I got a value of 20200122-200000 <tel:20200122-200000>.
When I used a filename of "wdgespcom/04-wdgespcom_20200122-020000.tsv.gz" I got a value of 20200122-070000, which I believe is correct.
Perhaps I misunderstood what you're trying to do? What version of NiFi are you using?
Thanks
-Mark
> On Jan 23, 2020, at 6:06 PM, Chowdhury, Rifat <Ri...@disney.com> wrote:
>
> Hi,
>
> I have a use case where I am storing a variable called “origin_timezone”. I am trying to convert a String which is originally in EST timezone to UTC time Zone. In my update Attribute, I write the following variable:
>
> "America/New_York"
>
> UTCDateString = ${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", ${origin_timezone}):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
>
> does not properly convert the String
>
> UTCDateString =${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", "America/New_York"):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
>
>
> while when I hardcode it to “America/New_York” it works perfectly. My use case is that I want to dynamically update UTCDateString so that I may convert any timeZone to UTC not just EST? How may I accomplish this?
>
> Example of what I am trying to do…
>
> Correct response:
> UTCDateString
> 20200122-200000 <tel:20200122-200000>
> filename
> wdgespcom/06-wdgespcom_20200122-150000.tsv.gz
>
>
> Incorrect Response:
> UTCDateString
> 20200122-020000 <tel:20200122-020000>
> filename
> wdgespcom/04-wdgespcom_20200122-020000.tsv.gz
>
>
> Rifat Chowdhury
> Software Engineer, Data Platforms
> <image001.png>
> THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
> Rifat.Chowdhury@disney.com <ma...@disney.com>