You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chandu Koripella <ck...@starbucks.com> on 2015/12/04 02:20:34 UTC
Content replacement
HI,
I am very new to the nifi and evaluating to use to do simple transformations. I am trying to convert date format from ddMMMyyyy to yyyy/mm/dd using replace text option. I don't find many options to do content replacement in the documentation. it is more focused for attribute manipulation.
[cid:image001.png@01D12DED.B5C91B90]
Hsere is my simple file format. how do I transform the data in col3? I appricaiate any help.
[cid:image003.png@01D12DEE.850E0550]
Thanks,
Chandu
Re: Content replacement
Posted by Mark Payne <ma...@hotmail.com>.
Chandu,
The release candidate is being voted on right now. Should come out Friday afternoon, barring any unexpected problems.
Thanks
-Mark
Sent from my iPhone
> On Dec 9, 2015, at 6:03 PM, Chandu Koripella <ck...@starbucks.com> wrote:
>
> Thank you Mark, This is great help. Could you please let me know when is version 0.4.0 coming out?
>
> Thanks,
> Chandu
>
>
> From: Mark Payne [mailto:markap14@hotmail.com]
> Sent: Friday, December 04, 2015 2:05 PM
> To: users@nifi.apache.org
> Subject: Re: Content replacement
>
> Chandru,
>
> This ticket has now been merged to master. This feature will be available in 0.4.0, which should be out next week. If you
> prefer to build from source, it is now available for use from master. You should be able to now configure ReplaceText to
> easily replace the values:
>
> <image001.png>
>
>
> Here, we use the Search value of (.*?),(.*?),(\d+.*)
> So the first Capturing Group gets the first column, the second Capturing Group gets the second column, and the third Capturing Group gets the third column (but only if it starts with a digit, so this won't match the header line and the header line will remain unchanged).
>
> For the Replacement Value, we use: $1,$2,${ '$3':toDate( 'ddMMMyyyy'):format('yyyy/MM/dd') }
> So here we are using back references to replace the line with the first two columns, followed by an Expression Language Expression that parses the third back reference (the third column).
> Since the variable that we want to reference is named $3, we need to enclose it in quotes to escape the name because it begins with a non-alpha character. We can then call any Expression Language
> Expression that we want. So toDate() can be used to parse the string as Date and then we can use format() to format that date as a string in a different format.
>
> You can also see the template available on the NIFI-1249 ticket, also attached for convenience here, but I don't know if the apache mailing list will let the template through.
>
> This was a great addition to NiFi that I think will help out in a lot of cases - thanks for reaching out to us on this!
> Please let us know if you have issues getting this to work, or if you have any further questions. We're happy to help however we can.
>
> Thanks
> -Mark
>
>
>
>
>
> On Dec 4, 2015, at 3:44 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Chandru,
>
> Correct what you're trying doesn't work just yet. But once this [1] is reviewed and pushed to master we should have your original request covered quite nicely.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1249
>
> Thanks
> Joe
>
> On Fri, Dec 4, 2015 at 3:26 PM, Chandu Koripella <ck...@starbucks.com> wrote:
> Thanks Joe,
>
>
>
> I believe I tried this option earlier. As you notice I have 4 groups in the regular expression. I tried with all four groups. Nothing gets replaced in the put file.
>
>
>
> Group4 is 4 digit year, group3 is 3 digit month, group2 is date and group1 is ddMMMyyyy
>
>
>
>
>
> <image001.png>
>
>
>
> From: Joe Witt [mailto:joe.witt@gmail.com]
> Sent: Thursday, December 03, 2015 7:00 PM
> To: users@nifi.apache.org
> Subject: Re: Content replacement
>
>
>
> Your use case description was excellent and helped expose a powerful and simple feature enhancement.
>
>
>
> https://issues.apache.org/jira/browse/NIFI-1249
>
>
>
> Thanks
>
> Joe
>
>
>
> On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Chandu,
>
>
>
> In building a template to support this I'm finding it more involved than desired. If I get that done will post it unless someone else does so sooner. However, the only reason it is more involved than I'd like is that the ReplaceText processor does almost everything we need here. It can be in line mode to do the eval/replace on a per line basis and it can extract groups from the matching line and use those groups to format a new output. However, the difficulty in your example is that we cannot take that date string and convert it to a date and then format it back to a string (right now). So will take a look at how we could better support the use case as well.
>
>
>
> Something else of note is that there are a couple of folks working on scripting processors so you could break into a convenient script to help tackle some cases like this as well.
>
>
>
> Will dig a bit more and respond.
>
>
>
> Thanks
>
> Joe
>
>
>
> On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com> wrote:
>
> HI,
>
>
>
> I am very new to the nifi and evaluating to use to do simple transformations. I am trying to convert date format from ddMMMyyyy to yyyy/mm/dd using replace text option. I don’t find many options to do content replacement in the documentation. it is more focused for attribute manipulation.
>
>
>
>
>
> <image004.png>
>
> Hsere is my simple file format. how do I transform the data in col3? I appricaiate any help.
>
>
>
> <image005.png>
>
>
>
> Thanks,
>
> Chandu
>
>
>
>
>
>
>
RE: Content replacement
Posted by Chandu Koripella <ck...@starbucks.com>.
Thank you Mark, This is great help. Could you please let me know when is version 0.4.0 coming out?
Thanks,
Chandu
From: Mark Payne [mailto:markap14@hotmail.com]
Sent: Friday, December 04, 2015 2:05 PM
To: users@nifi.apache.org
Subject: Re: Content replacement
Chandru,
This ticket has now been merged to master. This feature will be available in 0.4.0, which should be out next week. If you
prefer to build from source, it is now available for use from master. You should be able to now configure ReplaceText to
easily replace the values:
[cid:image001.png@01D13292.85081B10]
Here, we use the Search value of (.*?),(.*?),(\d+.*)
So the first Capturing Group gets the first column, the second Capturing Group gets the second column, and the third Capturing Group gets the third column (but only if it starts with a digit, so this won't match the header line and the header line will remain unchanged).
For the Replacement Value, we use: $1,$2,${ '$3':toDate( 'ddMMMyyyy'):format('yyyy/MM/dd') }
So here we are using back references to replace the line with the first two columns, followed by an Expression Language Expression that parses the third back reference (the third column).
Since the variable that we want to reference is named $3, we need to enclose it in quotes to escape the name because it begins with a non-alpha character. We can then call any Expression Language
Expression that we want. So toDate() can be used to parse the string as Date and then we can use format() to format that date as a string in a different format.
You can also see the template available on the NIFI-1249 ticket, also attached for convenience here, but I don't know if the apache mailing list will let the template through.
This was a great addition to NiFi that I think will help out in a lot of cases - thanks for reaching out to us on this!
Please let us know if you have issues getting this to work, or if you have any further questions. We're happy to help however we can.
Thanks
-Mark
On Dec 4, 2015, at 3:44 PM, Joe Witt <jo...@gmail.com>> wrote:
Chandru,
Correct what you're trying doesn't work just yet. But once this [1] is reviewed and pushed to master we should have your original request covered quite nicely.
[1] https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Fri, Dec 4, 2015 at 3:26 PM, Chandu Koripella <ck...@starbucks.com>> wrote:
Thanks Joe,
I believe I tried this option earlier. As you notice I have 4 groups in the regular expression. I tried with all four groups. Nothing gets replaced in the put file.
Group4 is 4 digit year, group3 is 3 digit month, group2 is date and group1 is ddMMMyyyy
<image001.png>
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Thursday, December 03, 2015 7:00 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Content replacement
Your use case description was excellent and helped expose a powerful and simple feature enhancement.
https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <jo...@gmail.com>> wrote:
Chandu,
In building a template to support this I'm finding it more involved than desired. If I get that done will post it unless someone else does so sooner. However, the only reason it is more involved than I'd like is that the ReplaceText processor does almost everything we need here. It can be in line mode to do the eval/replace on a per line basis and it can extract groups from the matching line and use those groups to format a new output. However, the difficulty in your example is that we cannot take that date string and convert it to a date and then format it back to a string (right now). So will take a look at how we could better support the use case as well.
Something else of note is that there are a couple of folks working on scripting processors so you could break into a convenient script to help tackle some cases like this as well.
Will dig a bit more and respond.
Thanks
Joe
On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com>> wrote:
HI,
I am very new to the nifi and evaluating to use to do simple transformations. I am trying to convert date format from ddMMMyyyy to yyyy/mm/dd using replace text option. I don’t find many options to do content replacement in the documentation. it is more focused for attribute manipulation.
<image004.png>
Hsere is my simple file format. how do I transform the data in col3? I appricaiate any help.
<image005.png>
Thanks,
Chandu
RE: Content replacement
Posted by Chandu Koripella <ck...@starbucks.com>.
That worked than you Joe
From: Joe Percivall [mailto:joepercivall@yahoo.com]
Sent: Tuesday, December 15, 2015 10:57 AM
To: users@nifi.apache.org
Subject: Re: Content replacement
Hey Chandu,
You messed up bit in the config of your ReplaceText processor. You should be running "Line-by-line" mode so each line is treated separately.
Also your search value is a bit messed up. It was trying to use capture groups in it ("$2"), it wasn't successfully handling an initial date of only 1 digit ("\d{2}"), it wasn't configured to match the spaces in column one and you need to either need explicitly make groups "non-capture groups" using "?:" or don't have them as capture groups. This search value works for me:
((?:\d{1,2}) (?:[a-zA-Z]{3}) (?:\d{4})),(.*),(.*)
Your replace value looks good though.
Hope that helps,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com<ma...@yahoo.com>
On Monday, December 14, 2015 5:21 PM, Chandu Koripella <ck...@starbucks.com>> wrote:
Hi Mark,
I am testing replace test feature in 0.4.0. I don’t see it is working. I tried with couple of the different columns. None of them aren’t replacing the text. can you please take a look?
[cid:image001.png@01D137EF.653971F0]
Thanks,
Chandu
From: Mark Payne [mailto:markap14@hotmail.com]
Sent: Friday, December 04, 2015 2:05 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Content replacement
Chandru,
This ticket has now been merged to master. This feature will be available in 0.4.0, which should be out next week. If you
prefer to build from source, it is now available for use from master. You should be able to now configure ReplaceText to
easily replace the values:
[cid:image002.png@01D137EF.653971F0]
Here, we use the Search value of (.*?),(.*?),(\d+.*)
So the first Capturing Group gets the first column, the second Capturing Group gets the second column, and the third Capturing Group gets the third column (but only if it starts with a digit, so this won't match the header line and the header line will remain unchanged).
For the Replacement Value, we use: $1,$2,${ '$3':toDate( 'ddMMMyyyy'):format('yyyy/MM/dd') }
So here we are using back references to replace the line with the first two columns, followed by an Expression Language Expression that parses the third back reference (the third column).
Since the variable that we want to reference is named $3, we need to enclose it in quotes to escape the name because it begins with a non-alpha character. We can then call any Expression Language
Expression that we want. So toDate() can be used to parse the string as Date and then we can use format() to format that date as a string in a different format.
You can also see the template available on the NIFI-1249 ticket, also attached for convenience here, but I don't know if the apache mailing list will let the template through.
This was a great addition to NiFi that I think will help out in a lot of cases - thanks for reaching out to us on this!
Please let us know if you have issues getting this to work, or if you have any further questions. We're happy to help however we can.
Thanks
-Mark
On Dec 4, 2015, at 3:44 PM, Joe Witt <jo...@gmail.com>> wrote:
Chandru,
Correct what you're trying doesn't work just yet. But once this [1] is reviewed and pushed to master we should have your original request covered quite nicely.
[1] https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Fri, Dec 4, 2015 at 3:26 PM, Chandu Koripella <ck...@starbucks.com>> wrote:
Thanks Joe,
I believe I tried this option earlier. As you notice I have 4 groups in the regular expression. I tried with all four groups. Nothing gets replaced in the put file.
Group4 is 4 digit year, group3 is 3 digit month, group2 is date and group1 is ddMMMyyyy
<image001.png>
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Thursday, December 03, 2015 7:00 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Content replacement
Your use case description was excellent and helped expose a powerful and simple feature enhancement.
https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <jo...@gmail.com>> wrote:
Chandu,
In building a template to support this I'm finding it more involved than desired. If I get that done will post it unless someone else does so sooner. However, the only reason it is more involved than I'd like is that the ReplaceText processor does almost everything we need here. It can be in line mode to do the eval/replace on a per line basis and it can extract groups from the matching line and use those groups to format a new output. However, the difficulty in your example is that we cannot take that date string and convert it to a date and then format it back to a string (right now). So will take a look at how we could better support the use case as well.
Something else of note is that there are a couple of folks working on scripting processors so you could break into a convenient script to help tackle some cases like this as well.
Will dig a bit more and respond.
Thanks
Joe
On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com>> wrote:
HI,
I am very new to the nifi and evaluating to use to do simple transformations. I am trying to convert date format from ddMMMyyyy to yyyy/mm/dd using replace text option. I don’t find many options to do content replacement in the documentation. it is more focused for attribute manipulation.
<image004.png>
Hsere is my simple file format. how do I transform the data in col3? I appricaiate any help.
<image005.png>
Thanks,
Chandu
Re: Content replacement
Posted by Joe Percivall <jo...@yahoo.com>.
Hey Chandu,
You messed up bit in the config of your ReplaceText processor. You should be running "Line-by-line" mode so each line is treated separately.
Also your search value is a bit messed up. It was trying to use capture groups in it ("$2"), it wasn't successfully handling an initial date of only 1 digit ("\d{2}"), it wasn't configured to match the spaces in column one and you need to either need explicitly make groups "non-capture groups" using "?:" or don't have them as capture groups. This search value works for me:
((?:\d{1,2}) (?:[a-zA-Z]{3}) (?:\d{4})),(.*),(.*)
Your replace value looks good though.
Hope that helps,Joe
- - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joepercivall@yahoo.com
On Monday, December 14, 2015 5:21 PM, Chandu Koripella <ck...@starbucks.com> wrote:
#yiv5326222985 #yiv5326222985 -- _filtered #yiv5326222985 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv5326222985 {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}#yiv5326222985 #yiv5326222985 p.yiv5326222985MsoNormal, #yiv5326222985 li.yiv5326222985MsoNormal, #yiv5326222985 div.yiv5326222985MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv5326222985 a:link, #yiv5326222985 span.yiv5326222985MsoHyperlink {color:blue;text-decoration:underline;}#yiv5326222985 a:visited, #yiv5326222985 span.yiv5326222985MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv5326222985 p.yiv5326222985MsoAcetate, #yiv5326222985 li.yiv5326222985MsoAcetate, #yiv5326222985 div.yiv5326222985MsoAcetate {margin:0in;margin-bottom:.0001pt;font-size:8.0pt;}#yiv5326222985 p.yiv5326222985xmsonormal, #yiv5326222985 li.yiv5326222985xmsonormal, #yiv5326222985 div.yiv5326222985xmsonormal {margin-right:0in;margin-left:0in;font-size:12.0pt;}#yiv5326222985 span.yiv5326222985BalloonTextChar {}#yiv5326222985 span.yiv5326222985EmailStyle20 {color:#1F497D;}#yiv5326222985 .yiv5326222985MsoChpDefault {font-size:10.0pt;} _filtered #yiv5326222985 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv5326222985 div.yiv5326222985WordSection1 {}#yiv5326222985 Hi Mark, I am testing replace test feature in 0.4.0. I don’t see it is working. I tried with couple of the different columns. None of them aren’t replacing the text. can you please take a look? Thanks, Chandu From: Mark Payne [mailto:markap14@hotmail.com]
Sent: Friday, December 04, 2015 2:05 PM
To: users@nifi.apache.org
Subject: Re: Content replacement Chandru, This ticket has now been merged to master. This feature will be available in 0.4.0, which should be out next week. If you prefer to build from source, it is now available for use from master. You should be able to now configure ReplaceText to easily replace the values: Here, we use the Search value of (.*?),(.*?),(\d+.*) So the first Capturing Group gets the first column, the second Capturing Group gets the second column, and the third Capturing Group gets the third column (but only if it starts with a digit, so this won't match the header line and the header line will remain unchanged). For the Replacement Value, we use: $1,$2,${ '$3':toDate( 'ddMMMyyyy'):format('yyyy/MM/dd') } So here we are using back references to replace the line with the first two columns, followed by an Expression Language Expression that parses the third back reference (the third column). Since the variable that we want to reference is named $3, we need to enclose it in quotes to escape the name because it begins with a non-alpha character. We can then call any Expression Language Expression that we want. So toDate() can be used to parse the string as Date and then we can use format() to format that date as a string in a different format. You can also see the template available on the NIFI-1249 ticket, also attached for convenience here, but I don't know if the apache mailing list will let the template through. This was a great addition to NiFi that I think will help out in a lot of cases - thanks for reaching out to us on this! Please let us know if you have issues getting this to work, or if you have any further questions. We're happy to help however we can. Thanks -Mark
On Dec 4, 2015, at 3:44 PM, Joe Witt <jo...@gmail.com> wrote: Chandru, Correct what you're trying doesn't work just yet. But once this [1] is reviewed and pushed to master we should have your original request covered quite nicely. [1] https://issues.apache.org/jira/browse/NIFI-1249 Thanks Joe On Fri, Dec 4, 2015 at 3:26 PM, Chandu Koripella <ck...@starbucks.com> wrote: Thanks Joe, I believe I tried this option earlier. As you notice I have 4 groups in the regular expression. I tried with all four groups. Nothing gets replaced in the put file. Group4 is 4 digit year, group3 is 3 digit month, group2 is date and group1 is ddMMMyyyy <image001.png> From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Thursday, December 03, 2015 7:00 PM
To: users@nifi.apache.org
Subject: Re: Content replacement Your use case description was excellent and helped expose a powerful and simple feature enhancement. https://issues.apache.org/jira/browse/NIFI-1249 Thanks Joe On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <jo...@gmail.com> wrote: Chandu, In building a template to support this I'm finding it more involved than desired. If I get that done will post it unless someone else does so sooner. However, the only reason it is more involved than I'd like is that the ReplaceText processor does almost everything we need here. It can be in line mode to do the eval/replace on a per line basis and it can extract groups from the matching line and use those groups to format a new output. However, the difficulty in your example is that we cannot take that date string and convert it to a date and then format it back to a string (right now). So will take a look at how we could better support the use case as well. Something else of note is that there are a couple of folks working on scripting processors so you could break into a convenient script to help tackle some cases like this as well. Will dig a bit more and respond. Thanks Joe On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com> wrote: HI, I am very new to the nifi and evaluating to use to do simple transformations. I am trying to convert date format from ddMMMyyyy to yyyy/mm/dd using replace text option. I don’t find many options to do content replacement in the documentation. it is more focused for attribute manipulation. <image004.png> Hsere is my simple file format. how do I transform the data in col3? I appricaiate any help. <image005.png> Thanks, Chandu
RE: Content replacement
Posted by Chandu Koripella <ck...@starbucks.com>.
Hi Mark,
I am testing replace test feature in 0.4.0. I don’t see it is working. I tried with couple of the different columns. None of them aren’t replacing the text. can you please take a look?
[cid:image002.png@01D1367A.5001F2B0]
Thanks,
Chandu
From: Mark Payne [mailto:markap14@hotmail.com]
Sent: Friday, December 04, 2015 2:05 PM
To: users@nifi.apache.org
Subject: Re: Content replacement
Chandru,
This ticket has now been merged to master. This feature will be available in 0.4.0, which should be out next week. If you
prefer to build from source, it is now available for use from master. You should be able to now configure ReplaceText to
easily replace the values:
[cid:image001.png@01D13679.BFEA2170]
Here, we use the Search value of (.*?),(.*?),(\d+.*)
So the first Capturing Group gets the first column, the second Capturing Group gets the second column, and the third Capturing Group gets the third column (but only if it starts with a digit, so this won't match the header line and the header line will remain unchanged).
For the Replacement Value, we use: $1,$2,${ '$3':toDate( 'ddMMMyyyy'):format('yyyy/MM/dd') }
So here we are using back references to replace the line with the first two columns, followed by an Expression Language Expression that parses the third back reference (the third column).
Since the variable that we want to reference is named $3, we need to enclose it in quotes to escape the name because it begins with a non-alpha character. We can then call any Expression Language
Expression that we want. So toDate() can be used to parse the string as Date and then we can use format() to format that date as a string in a different format.
You can also see the template available on the NIFI-1249 ticket, also attached for convenience here, but I don't know if the apache mailing list will let the template through.
This was a great addition to NiFi that I think will help out in a lot of cases - thanks for reaching out to us on this!
Please let us know if you have issues getting this to work, or if you have any further questions. We're happy to help however we can.
Thanks
-Mark
On Dec 4, 2015, at 3:44 PM, Joe Witt <jo...@gmail.com>> wrote:
Chandru,
Correct what you're trying doesn't work just yet. But once this [1] is reviewed and pushed to master we should have your original request covered quite nicely.
[1] https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Fri, Dec 4, 2015 at 3:26 PM, Chandu Koripella <ck...@starbucks.com>> wrote:
Thanks Joe,
I believe I tried this option earlier. As you notice I have 4 groups in the regular expression. I tried with all four groups. Nothing gets replaced in the put file.
Group4 is 4 digit year, group3 is 3 digit month, group2 is date and group1 is ddMMMyyyy
<image001.png>
From: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Sent: Thursday, December 03, 2015 7:00 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Content replacement
Your use case description was excellent and helped expose a powerful and simple feature enhancement.
https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <jo...@gmail.com>> wrote:
Chandu,
In building a template to support this I'm finding it more involved than desired. If I get that done will post it unless someone else does so sooner. However, the only reason it is more involved than I'd like is that the ReplaceText processor does almost everything we need here. It can be in line mode to do the eval/replace on a per line basis and it can extract groups from the matching line and use those groups to format a new output. However, the difficulty in your example is that we cannot take that date string and convert it to a date and then format it back to a string (right now). So will take a look at how we could better support the use case as well.
Something else of note is that there are a couple of folks working on scripting processors so you could break into a convenient script to help tackle some cases like this as well.
Will dig a bit more and respond.
Thanks
Joe
On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com>> wrote:
HI,
I am very new to the nifi and evaluating to use to do simple transformations. I am trying to convert date format from ddMMMyyyy to yyyy/mm/dd using replace text option. I don’t find many options to do content replacement in the documentation. it is more focused for attribute manipulation.
<image004.png>
Hsere is my simple file format. how do I transform the data in col3? I appricaiate any help.
<image005.png>
Thanks,
Chandu
Re: Content replacement
Posted by Mark Payne <ma...@hotmail.com>.
Chandru,
This ticket has now been merged to master. This feature will be available in 0.4.0, which should be out next week. If you
prefer to build from source, it is now available for use from master. You should be able to now configure ReplaceText to
easily replace the values:
Here, we use the Search value of (.*?),(.*?),(\d+.*)
So the first Capturing Group gets the first column, the second Capturing Group gets the second column, and the third Capturing Group gets the third column (but only if it starts with a digit, so this won't match the header line and the header line will remain unchanged).
For the Replacement Value, we use: $1,$2,${ '$3':toDate( 'ddMMMyyyy'):format('yyyy/MM/dd') }
So here we are using back references to replace the line with the first two columns, followed by an Expression Language Expression that parses the third back reference (the third column).
Since the variable that we want to reference is named $3, we need to enclose it in quotes to escape the name because it begins with a non-alpha character. We can then call any Expression Language
Expression that we want. So toDate() can be used to parse the string as Date and then we can use format() to format that date as a string in a different format.
You can also see the template available on the NIFI-1249 ticket, also attached for convenience here, but I don't know if the apache mailing list will let the template through.
This was a great addition to NiFi that I think will help out in a lot of cases - thanks for reaching out to us on this!
Please let us know if you have issues getting this to work, or if you have any further questions. We're happy to help however we can.
Thanks
-Mark
> On Dec 4, 2015, at 3:44 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Chandru,
>
> Correct what you're trying doesn't work just yet. But once this [1] is reviewed and pushed to master we should have your original request covered quite nicely.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1249 <https://issues.apache.org/jira/browse/NIFI-1249>
>
> Thanks
> Joe
>
> On Fri, Dec 4, 2015 at 3:26 PM, Chandu Koripella <ckoripel@starbucks.com <ma...@starbucks.com>> wrote:
> Thanks Joe,
>
>
>
> I believe I tried this option earlier. As you notice I have 4 groups in the regular expression. I tried with all four groups. Nothing gets replaced in the put file.
>
>
>
> Group4 is 4 digit year, group3 is 3 digit month, group2 is date and group1 is ddMMMyyyy
>
>
>
>
>
> <image001.png>
>
>
>
> From: Joe Witt [mailto:joe.witt@gmail.com <ma...@gmail.com>]
> Sent: Thursday, December 03, 2015 7:00 PM
> To: users@nifi.apache.org <ma...@nifi.apache.org>
> Subject: Re: Content replacement
>
>
>
> Your use case description was excellent and helped expose a powerful and simple feature enhancement.
>
>
>
> https://issues.apache.org/jira/browse/NIFI-1249 <https://issues.apache.org/jira/browse/NIFI-1249>
>
>
> Thanks
>
> Joe
>
>
>
> On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>
> Chandu,
>
>
>
> In building a template to support this I'm finding it more involved than desired. If I get that done will post it unless someone else does so sooner. However, the only reason it is more involved than I'd like is that the ReplaceText processor does almost everything we need here. It can be in line mode to do the eval/replace on a per line basis and it can extract groups from the matching line and use those groups to format a new output. However, the difficulty in your example is that we cannot take that date string and convert it to a date and then format it back to a string (right now). So will take a look at how we could better support the use case as well.
>
>
>
> Something else of note is that there are a couple of folks working on scripting processors so you could break into a convenient script to help tackle some cases like this as well.
>
>
>
> Will dig a bit more and respond.
>
>
>
> Thanks
>
> Joe
>
>
>
> On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ckoripel@starbucks.com <ma...@starbucks.com>> wrote:
>
> HI,
>
>
>
> I am very new to the nifi and evaluating to use to do simple transformations. I am trying to convert date format from ddMMMyyyy to yyyy/mm/dd using replace text option. I don’t find many options to do content replacement in the documentation. it is more focused for attribute manipulation.
>
>
>
>
>
> <image004.png>
>
> Hsere is my simple file format. how do I transform the data in col3? I appricaiate any help.
>
>
>
> <image005.png>
>
>
>
> Thanks,
>
> Chandu
>
>
>
>
>
>
Re: Content replacement
Posted by Joe Witt <jo...@gmail.com>.
Chandru,
Correct what you're trying doesn't work just yet. But once this [1] is
reviewed and pushed to master we should have your original request covered
quite nicely.
[1] https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Fri, Dec 4, 2015 at 3:26 PM, Chandu Koripella <ck...@starbucks.com>
wrote:
> Thanks Joe,
>
>
>
> I believe I tried this option earlier. As you notice I have 4 groups in
> the regular expression. I tried with all four groups. Nothing gets replaced
> in the put file.
>
>
>
> Group4 is 4 digit year, group3 is 3 digit month, group2 is date and group1
> is ddMMMyyyy
>
>
>
>
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Thursday, December 03, 2015 7:00 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Content replacement
>
>
>
> Your use case description was excellent and helped expose a powerful and
> simple feature enhancement.
>
>
>
> https://issues.apache.org/jira/browse/NIFI-1249
>
>
>
> Thanks
>
> Joe
>
>
>
> On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Chandu,
>
>
>
> In building a template to support this I'm finding it more involved than
> desired. If I get that done will post it unless someone else does so
> sooner. However, the only reason it is more involved than I'd like is that
> the ReplaceText processor does almost everything we need here. It can be
> in line mode to do the eval/replace on a per line basis and it can extract
> groups from the matching line and use those groups to format a new output.
> However, the difficulty in your example is that we cannot take that date
> string and convert it to a date and then format it back to a string (right
> now). So will take a look at how we could better support the use case as
> well.
>
>
>
> Something else of note is that there are a couple of folks working on
> scripting processors so you could break into a convenient script to help
> tackle some cases like this as well.
>
>
>
> Will dig a bit more and respond.
>
>
>
> Thanks
>
> Joe
>
>
>
> On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com>
> wrote:
>
> HI,
>
>
>
> I am very new to the nifi and evaluating to use to do simple
> transformations. I am trying to convert date format from ddMMMyyyy to
> yyyy/mm/dd using replace text option. I don’t find many options to do
> content replacement in the documentation. it is more focused for attribute
> manipulation.
>
>
>
>
>
> Hsere is my simple file format. how do I transform the data in col3? I
> appricaiate any help.
>
>
>
>
>
> Thanks,
>
> Chandu
>
>
>
>
>
RE: Content replacement
Posted by Chandu Koripella <ck...@starbucks.com>.
Thanks Joe,
I believe I tried this option earlier. As you notice I have 4 groups in the regular expression. I tried with all four groups. Nothing gets replaced in the put file.
Group4 is 4 digit year, group3 is 3 digit month, group2 is date and group1 is ddMMMyyyy
[cid:image001.png@01D12E8E.5BE4CC90]
From: Joe Witt [mailto:joe.witt@gmail.com]
Sent: Thursday, December 03, 2015 7:00 PM
To: users@nifi.apache.org
Subject: Re: Content replacement
Your use case description was excellent and helped expose a powerful and simple feature enhancement.
https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <jo...@gmail.com>> wrote:
Chandu,
In building a template to support this I'm finding it more involved than desired. If I get that done will post it unless someone else does so sooner. However, the only reason it is more involved than I'd like is that the ReplaceText processor does almost everything we need here. It can be in line mode to do the eval/replace on a per line basis and it can extract groups from the matching line and use those groups to format a new output. However, the difficulty in your example is that we cannot take that date string and convert it to a date and then format it back to a string (right now). So will take a look at how we could better support the use case as well.
Something else of note is that there are a couple of folks working on scripting processors so you could break into a convenient script to help tackle some cases like this as well.
Will dig a bit more and respond.
Thanks
Joe
On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com>> wrote:
HI,
I am very new to the nifi and evaluating to use to do simple transformations. I am trying to convert date format from ddMMMyyyy to yyyy/mm/dd using replace text option. I don’t find many options to do content replacement in the documentation. it is more focused for attribute manipulation.
[cid:image004.png@01D12E8D.55EF0810]
Hsere is my simple file format. how do I transform the data in col3? I appricaiate any help.
[cid:image005.png@01D12E8D.55EF0810]
Thanks,
Chandu
Re: Content replacement
Posted by Joe Witt <jo...@gmail.com>.
Your use case description was excellent and helped expose a powerful and
simple feature enhancement.
https://issues.apache.org/jira/browse/NIFI-1249
Thanks
Joe
On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <jo...@gmail.com> wrote:
> Chandu,
>
> In building a template to support this I'm finding it more involved than
> desired. If I get that done will post it unless someone else does so
> sooner. However, the only reason it is more involved than I'd like is that
> the ReplaceText processor does almost everything we need here. It can be
> in line mode to do the eval/replace on a per line basis and it can extract
> groups from the matching line and use those groups to format a new output.
> However, the difficulty in your example is that we cannot take that date
> string and convert it to a date and then format it back to a string (right
> now). So will take a look at how we could better support the use case as
> well.
>
> Something else of note is that there are a couple of folks working on
> scripting processors so you could break into a convenient script to help
> tackle some cases like this as well.
>
> Will dig a bit more and respond.
>
> Thanks
> Joe
>
> On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com>
> wrote:
>
>> HI,
>>
>>
>>
>> I am very new to the nifi and evaluating to use to do simple
>> transformations. I am trying to convert date format from ddMMMyyyy to
>> yyyy/mm/dd using replace text option. I don’t find many options to do
>> content replacement in the documentation. it is more focused for attribute
>> manipulation.
>>
>>
>>
>>
>>
>> Hsere is my simple file format. how do I transform the data in col3? I
>> appricaiate any help.
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Chandu
>>
>
>
Re: Content replacement
Posted by Joe Witt <jo...@gmail.com>.
Chandu,
In building a template to support this I'm finding it more involved than
desired. If I get that done will post it unless someone else does so
sooner. However, the only reason it is more involved than I'd like is that
the ReplaceText processor does almost everything we need here. It can be
in line mode to do the eval/replace on a per line basis and it can extract
groups from the matching line and use those groups to format a new output.
However, the difficulty in your example is that we cannot take that date
string and convert it to a date and then format it back to a string (right
now). So will take a look at how we could better support the use case as
well.
Something else of note is that there are a couple of folks working on
scripting processors so you could break into a convenient script to help
tackle some cases like this as well.
Will dig a bit more and respond.
Thanks
Joe
On Thu, Dec 3, 2015 at 8:20 PM, Chandu Koripella <ck...@starbucks.com>
wrote:
> HI,
>
>
>
> I am very new to the nifi and evaluating to use to do simple
> transformations. I am trying to convert date format from ddMMMyyyy to
> yyyy/mm/dd using replace text option. I don’t find many options to do
> content replacement in the documentation. it is more focused for attribute
> manipulation.
>
>
>
>
>
> Hsere is my simple file format. how do I transform the data in col3? I
> appricaiate any help.
>
>
>
>
>
> Thanks,
>
> Chandu
>