You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Chowdhury, Rifat" <Ri...@disney.com> on 2020/01/23 23:06:34 UTC

Nifi Expression Language Question

Hi,

I have a use case where I am storing a variable called “origin_timezone”. I am trying to convert a String which is originally in EST timezone to UTC time Zone. In my update Attribute, I write the following variable:

"America/New_York"

UTCDateString = ${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", ${origin_timezone}):toNumber():format("yyyyMMdd-HHmmss", "UTC")}

does not properly convert the String

UTCDateString =${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", "America/New_York"):toNumber():format("yyyyMMdd-HHmmss", "UTC")}


while when I hardcode it to “America/New_York” it works perfectly. My use case is that I want to dynamically update UTCDateString so that I may convert any timeZone to UTC not just EST? How may I accomplish this?

Example of what I am trying to do…

Correct response:
UTCDateString
20200122-200000
filename
wdgespcom/06-wdgespcom_20200122-150000.tsv.gz


Incorrect Response:
UTCDateString
20200122-020000
filename
wdgespcom/04-wdgespcom_20200122-020000.tsv.gz


Rifat Chowdhury
Software Engineer, Data Platforms
[signature_1999449656]
 THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>


Re: Nifi Expression Language Question

Posted by "Chowdhury, Rifat" <Ri...@disney.com>.
Hi Mark,

I am using Nifi 1.10 on an EKS (Kubernetes) cluster. I read in a lookup csv file store that csv file in Redis. That lookup file contains the names of suites, different suites are from different time zones. For example, suite A is from “America/New_York” and suite B is from “America/Los_Angeles”. Once I read the lookup data from RedisCache, and extract the time zone from the lookup data at that point I would store that timezone in a variable called “origin_timezone”.

For example: if the “origin_timezone” = “America/Los_Angeles”,  and the filename = "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" Then my UTCDateString = 20200122-230000<tel:20200122-200000>.

If the “origin_timezone” = “America/New_York”,  and the filename = "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" Then my UTCDateString = 20200122-200000<tel:20200122-200000>. But in my Nifi Template, using the “origin_timezone” variable does not properly extract the correct UTCDateString. I have attached my Nifi Template for reference.

Rifat Chowdhury
Software Engineer, Data Platforms
[signature_1999449656]
 THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>

From: Mark Payne <ma...@hotmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, January 23, 2020 at 6:30 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Cc: "Rudra, Anshuman X. -ND" <An...@espn.com>, "Loyot, Brendan" <Br...@espn.com>, "Nagwani, Dheeraj X. -ND" <dh...@disney.com>
Subject: Re: Nifi Expression Language Question

Rifat,

I tried the example here. I got the same response whether using "America/New_York" hardcoded or using ${origin_timezone}. When I used the filename "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" I got a value of 20200122-200000<tel:20200122-200000>.
When I used a filename of "wdgespcom/04-wdgespcom_20200122-020000.tsv.gz" I got a value of 20200122-070000, which I believe is correct.


Perhaps I misunderstood what you're trying to do? What version of NiFi are you using?


Thanks
-Mark








On Jan 23, 2020, at 6:06 PM, Chowdhury, Rifat <Ri...@disney.com>> wrote:

Hi,

I have a use case where I am storing a variable called “origin_timezone”. I am trying to convert a String which is originally in EST timezone to UTC time Zone. In my update Attribute, I write the following variable:

"America/New_York"

UTCDateString = ${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", ${origin_timezone}):toNumber():format("yyyyMMdd-HHmmss", "UTC")}

does not properly convert the String


UTCDateString =${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", "America/New_York"):toNumber():format("yyyyMMdd-HHmmss", "UTC")}


while when I hardcode it to “America/New_York” it works perfectly. My use case is that I want to dynamically update UTCDateString so that I may convert any timeZone to UTC not just EST? How may I accomplish this?

Example of what I am trying to do…

Correct response:
UTCDateString
20200122-200000<tel:20200122-200000>
filename
wdgespcom/06-wdgespcom_20200122-150000.tsv.gz


Incorrect Response:
UTCDateString
20200122-020000<tel:20200122-020000>
filename
wdgespcom/04-wdgespcom_20200122-020000.tsv.gz



Rifat Chowdhury
Software Engineer, Data Platforms
<image001.png>
 THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>


Re: Nifi Expression Language Question

Posted by "Chowdhury, Rifat" <Ri...@disney.com>.
Never Mind. The mistake I made was When I extracted the string, I also extracted the quotation marks…the Nifi ToDate function was parsing “"America/New_York"” instead of "America/New_York"… that’s why it worked when I hardcoded it 😊 Thanks Mark and Anshuman for your help. Silly on my part.


Rifat Chowdhury
Software Engineer, Data Platforms
[signature_1999449656]
 THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>


From: Mark Payne <ma...@hotmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, January 23, 2020 at 6:30 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Cc: "Rudra, Anshuman X. -ND" <An...@espn.com>, "Loyot, Brendan" <Br...@espn.com>, "Nagwani, Dheeraj X. -ND" <dh...@disney.com>
Subject: Re: Nifi Expression Language Question

Rifat,

I tried the example here. I got the same response whether using "America/New_York" hardcoded or using ${origin_timezone}. When I used the filename "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" I got a value of 20200122-200000<tel:20200122-200000>.
When I used a filename of "wdgespcom/04-wdgespcom_20200122-020000.tsv.gz" I got a value of 20200122-070000, which I believe is correct.


Perhaps I misunderstood what you're trying to do? What version of NiFi are you using?


Thanks
-Mark








On Jan 23, 2020, at 6:06 PM, Chowdhury, Rifat <Ri...@disney.com>> wrote:

Hi,

I have a use case where I am storing a variable called “origin_timezone”. I am trying to convert a String which is originally in EST timezone to UTC time Zone. In my update Attribute, I write the following variable:

"America/New_York"

UTCDateString = ${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", ${origin_timezone}):toNumber():format("yyyyMMdd-HHmmss", "UTC")}

does not properly convert the String


UTCDateString =${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", "America/New_York"):toNumber():format("yyyyMMdd-HHmmss", "UTC")}


while when I hardcode it to “America/New_York” it works perfectly. My use case is that I want to dynamically update UTCDateString so that I may convert any timeZone to UTC not just EST? How may I accomplish this?

Example of what I am trying to do…

Correct response:
UTCDateString
20200122-200000<tel:20200122-200000>
filename
wdgespcom/06-wdgespcom_20200122-150000.tsv.gz


Incorrect Response:
UTCDateString
20200122-020000<tel:20200122-020000>
filename
wdgespcom/04-wdgespcom_20200122-020000.tsv.gz



Rifat Chowdhury
Software Engineer, Data Platforms
<image001.png>
 THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
Rifat.Chowdhury@disney.com<ma...@disney.com>


Re: Nifi Expression Language Question

Posted by Mark Payne <ma...@hotmail.com>.
Rifat,

I tried the example here. I got the same response whether using "America/New_York" hardcoded or using ${origin_timezone}. When I used the filename "wdgespcom/06-wdgespcom_20200122-150000.tsv.gz" I got a value of 20200122-200000 <tel:20200122-200000>.
When I used a filename of "wdgespcom/04-wdgespcom_20200122-020000.tsv.gz" I got a value of 20200122-070000, which I believe is correct.

Perhaps I misunderstood what you're trying to do? What version of NiFi are you using?

Thanks
-Mark





> On Jan 23, 2020, at 6:06 PM, Chowdhury, Rifat <Ri...@disney.com> wrote:
> 
> Hi,
> 
> I have a use case where I am storing a variable called “origin_timezone”. I am trying to convert a String which is originally in EST timezone to UTC time Zone. In my update Attribute, I write the following variable:
>  
> "America/New_York"
>  
> UTCDateString = ${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", ${origin_timezone}):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
> 
> does not properly convert the String
> 
> UTCDateString =${filename:substringAfterLast('_'):substringBefore('.'):toDate("yyyyMMdd-HHmmss", "America/New_York"):toNumber():format("yyyyMMdd-HHmmss", "UTC")}
> 
> 
> while when I hardcode it to “America/New_York” it works perfectly. My use case is that I want to dynamically update UTCDateString so that I may convert any timeZone to UTC not just EST? How may I accomplish this?
> 
> Example of what I am trying to do…
> 
> Correct response:
> UTCDateString
> 20200122-200000 <tel:20200122-200000>
> filename
> wdgespcom/06-wdgespcom_20200122-150000.tsv.gz
> 
> 
> Incorrect Response:
> UTCDateString
> 20200122-020000 <tel:20200122-020000>
> filename
> wdgespcom/04-wdgespcom_20200122-020000.tsv.gz
> 
>  
> Rifat Chowdhury
> Software Engineer, Data Platforms
> <image001.png>
>  THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL
> Rifat.Chowdhury@disney.com <ma...@disney.com>