You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jake Kugel <ja...@yahoo.com> on 2016/07/22 03:18:36 UTC
Using ReplaceText to insert newline character into flowfile?
Hello,
Is it possible to use ReplaceText processor to insert a newline
character (0x0a) or other special characters into a flowfile? I've
tried putting \r and \n in the value of ReplaceText's Replacement Value
property, but they are inserted literally into the file (the Regex
Replace strategy will insert an 'n', and Literal Replace strategy will
insert '\n'). Is there another way to do it?
Thanks,
Jake
Re: Using ReplaceText to insert newline character into flowfile?
Posted by Andy LoPresto <al...@apache.org>.
Carl,
This is a great suggestion. I think you can improve the performance of your script by moving the StringBuilder outside of the loop (or even further, simply reading all the lines into a list and joining them with the delimiter you want):
import java.nio.charset.StandardCharsets
log.info("Add backslash R with Groovy!")
def flowFile = session.get()
if (!flowFile) return
flowFile = session.write(flowFile, {inputStream, outputStream ->
String allTheData = inputStream.readLines().join("\r\n")
log.info("addBackslashRFromMsg allTheData = " + allTheData)
outputStream.write(allTheData.getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
log.info("Add backslash R with Groovy all done! (before session.transfer) ")
session.transfer(flowFile, REL_SUCCESS)
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 2, 2016, at 8:06 PM, Jake Kugel <ja...@yahoo.com> wrote:
>
> Interesting, hadn't looked at using ExecuteScript before this but looks like ExecuteScript and Groovy would let me do what I need to do, I will try that approach. Thanks,
> Jake
>
> On 7/28/16 2:02 PM, Carl Berndt wrote:
>> Hey Jake,
>> I had a similar issue recently. I used this .groovy script :
>>
>> import java.nio.charset.StandardCharsets
>> log.info <http://log.info/>("Add backslash R with Groovy!") ;
>> def flowFile = session.get();
>> if (!flowFile) return;
>>
>> flowFile = session.putAttribute(flowFile, 'my-property', 'my-value');
>>
>> flowFile = session.write(flowFile, {inputStream, outputStream ->
>>
>> inputStream.eachLine { line ->
>> StringBuilder sb = new StringBuilder();
>> sb.append(line);
>> sb.append("\r\n");
>> String allTheData = sb.toString()
>> log.info <http://log.info/>("addBackslashRFromMsg allTheData = " + allTheData)
>> outputStream.write(allTheData.getBytes(StandardCharsets.UTF_8))
>> }
>>
>> } as StreamCallback)
>>
>> log.info <http://log.info/>("Add backslash R with Groovy all done! (before session.transfer) ") ;
>> session.transfer(flowFile, REL_SUCCESS)
>>
>> Carl
>>
>> On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <jake_kugel@yahoo.com <ma...@yahoo.com>> wrote:
>> Thanks, creating the newline in the replacement value property with shift-enter worked! The newline character (0x0a) was inserted into the flowfile.
>>
>> My use case is slightly more involved, I also need to insert carriage returns (0x0d) in some cases. But I think you've given me a direction to go, I'll see if I can copy/paste a carriage return character into the replacement value.
>>
>> Thanks again,
>> Jake
>>
>>
>> On 7/21/16 10:38 PM, Joe Witt wrote:
>> Jake,
>>
>> In the 'Replacement Value' you can tell it what you want then you can
>> hit "Shift-Enter" and it will insert the newline.
>>
>> So, lets say the input you're getting into this processor is a
>> flowfile containing a bunch of line seperated events. If you wanted
>> to insert a newline in between each line you could use a config for
>> ReplaceText such as
>>
>> Search value: (^.*$)
>> Replacement value: $1<hit shift enter and you'll see it put a newline>
>> Replacement strategy: Regex Replace
>> Evaluation Mode: Line by line
>>
>> With such a config lets say you passed in data that was
>>
>> some line 1
>> some line 2
>> some line 3
>>
>> Then after your processor you'd end up with
>>
>> some line 1
>>
>> some line 2
>>
>> some line 3
>>
>>
>> Does that get you where you want to be?
>>
>> Thanks
>> Joe
>>
>> On Thu, Jul 21, 2016 at 11:28 PM, Jake Kugel <jake_kugel@yahoo.com <ma...@yahoo.com>> wrote:
>> Thanks for the reply. I tried just now using \\n with the regex replace
>> strategy, and it put '\n' into the flowfile instead of 'n'.
>>
>> Jake
>>
>>
>> On 7/21/16 10:22 PM, Joe Witt wrote:
>>
>> Jake,
>>
>> Can you give \\n a try on the regex replace strategy? Let us know if
>> that does it.
>>
>> Thanks
>> Joe
>>
>> On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <jake_kugel@yahoo.com <ma...@yahoo.com>> wrote:
>>
>> Hello,
>>
>> Is it possible to use ReplaceText processor to insert a newline character
>> (0x0a) or other special characters into a flowfile? I've tried putting
>> \r
>> and \n in the value of ReplaceText's Replacement Value property, but they
>> are inserted literally into the file (the Regex Replace strategy will
>> insert
>> an 'n', and Literal Replace strategy will insert '\n'). Is there another
>> way to do it?
>>
>> Thanks,
>> Jake
>>
>
Re: Using ReplaceText to insert newline character into flowfile?
Posted by Jake Kugel <ja...@yahoo.com>.
Interesting, hadn't looked at using ExecuteScript before this but looks
like ExecuteScript and Groovy would let me do what I need to do, I will
try that approach. Thanks,
Jake
On 7/28/16 2:02 PM, Carl Berndt wrote:
> Hey Jake,
> I had a similar issue recently. I used this .groovy script :
>
> import java.nio.charset.StandardCharsets
> log.info <http://log.info>("Add backslash R with Groovy!") ;
> def flowFile = session.get();
> if (!flowFile) return;
>
> flowFile = session.putAttribute(flowFile, 'my-property', 'my-value');
>
> flowFile = session.write(flowFile, {inputStream, outputStream ->
>
> inputStream.eachLine { line ->
> StringBuilder sb = new StringBuilder();
> sb.append(line);
> sb.append("\r\n");
> String allTheData = sb.toString()
> log.info <http://log.info>("addBackslashRFromMsg allTheData = " +
> allTheData)
> outputStream.write(allTheData.getBytes(StandardCharsets.UTF_8))
> }
>
> } as StreamCallback)
>
> log.info <http://log.info>("Add backslash R with Groovy all done!
> (before session.transfer) ") ;
> session.transfer(flowFile, REL_SUCCESS)
>
> Carl
>
> On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <jake_kugel@yahoo.com
> <ma...@yahoo.com>> wrote:
>
> Thanks, creating the newline in the replacement value property
> with shift-enter worked! The newline character (0x0a) was
> inserted into the flowfile.
>
> My use case is slightly more involved, I also need to insert
> carriage returns (0x0d) in some cases. But I think you've given
> me a direction to go, I'll see if I can copy/paste a carriage
> return character into the replacement value.
>
> Thanks again,
> Jake
>
>
> On 7/21/16 10:38 PM, Joe Witt wrote:
>
> Jake,
>
> In the 'Replacement Value' you can tell it what you want then
> you can
> hit "Shift-Enter" and it will insert the newline.
>
> So, lets say the input you're getting into this processor is a
> flowfile containing a bunch of line seperated events. If you
> wanted
> to insert a newline in between each line you could use a
> config for
> ReplaceText such as
>
> Search value: (^.*$)
> Replacement value: $1<hit shift enter and you'll see it put a
> newline>
> Replacement strategy: Regex Replace
> Evaluation Mode: Line by line
>
> With such a config lets say you passed in data that was
>
> some line 1
> some line 2
> some line 3
>
> Then after your processor you'd end up with
>
> some line 1
>
> some line 2
>
> some line 3
>
>
> Does that get you where you want to be?
>
> Thanks
> Joe
>
> On Thu, Jul 21, 2016 at 11:28 PM, Jake Kugel
> <jake_kugel@yahoo.com <ma...@yahoo.com>> wrote:
>
> Thanks for the reply. I tried just now using \\n with the
> regex replace
> strategy, and it put '\n' into the flowfile instead of 'n'.
>
> Jake
>
>
> On 7/21/16 10:22 PM, Joe Witt wrote:
>
>
> Jake,
>
> Can you give \\n a try on the regex replace strategy?
> Let us know if
> that does it.
>
> Thanks
> Joe
>
> On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel
> <jake_kugel@yahoo.com <ma...@yahoo.com>>
> wrote:
>
>
> Hello,
>
> Is it possible to use ReplaceText processor to
> insert a newline character
> (0x0a) or other special characters into a
> flowfile? I've tried putting
> \r
> and \n in the value of ReplaceText's Replacement
> Value property, but they
> are inserted literally into the file (the Regex
> Replace strategy will
> insert
> an 'n', and Literal Replace strategy will insert
> '\n'). Is there another
> way to do it?
>
> Thanks,
> Jake
>
>
Re: Using ReplaceText to insert newline character into flowfile?
Posted by Carl Berndt <cs...@gmail.com>.
Hey Jake,
I had a similar issue recently. I used this .groovy script :
import java.nio.charset.StandardCharsets
log.info("Add backslash R with Groovy!") ;
def flowFile = session.get();
if (!flowFile) return;
flowFile = session.putAttribute(flowFile, 'my-property', 'my-value');
flowFile = session.write(flowFile, {inputStream, outputStream ->
inputStream.eachLine { line ->
StringBuilder sb = new StringBuilder();
sb.append(line);
sb.append("\r\n");
String allTheData = sb.toString()
log.info("addBackslashRFromMsg allTheData = " + allTheData)
outputStream.write(allTheData.getBytes(StandardCharsets.UTF_8))
}
} as StreamCallback)
log.info("Add backslash R with Groovy all done! (before session.transfer)
") ;
session.transfer(flowFile, REL_SUCCESS)
Carl
On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <ja...@yahoo.com> wrote:
> Thanks, creating the newline in the replacement value property with
> shift-enter worked! The newline character (0x0a) was inserted into the
> flowfile.
>
> My use case is slightly more involved, I also need to insert carriage
> returns (0x0d) in some cases. But I think you've given me a direction to
> go, I'll see if I can copy/paste a carriage return character into the
> replacement value.
>
> Thanks again,
> Jake
>
>
> On 7/21/16 10:38 PM, Joe Witt wrote:
>
>> Jake,
>>
>> In the 'Replacement Value' you can tell it what you want then you can
>> hit "Shift-Enter" and it will insert the newline.
>>
>> So, lets say the input you're getting into this processor is a
>> flowfile containing a bunch of line seperated events. If you wanted
>> to insert a newline in between each line you could use a config for
>> ReplaceText such as
>>
>> Search value: (^.*$)
>> Replacement value: $1<hit shift enter and you'll see it put a newline>
>> Replacement strategy: Regex Replace
>> Evaluation Mode: Line by line
>>
>> With such a config lets say you passed in data that was
>>
>> some line 1
>> some line 2
>> some line 3
>>
>> Then after your processor you'd end up with
>>
>> some line 1
>>
>> some line 2
>>
>> some line 3
>>
>>
>> Does that get you where you want to be?
>>
>> Thanks
>> Joe
>>
>> On Thu, Jul 21, 2016 at 11:28 PM, Jake Kugel <ja...@yahoo.com>
>> wrote:
>>
>>> Thanks for the reply. I tried just now using \\n with the regex replace
>>> strategy, and it put '\n' into the flowfile instead of 'n'.
>>>
>>> Jake
>>>
>>>
>>> On 7/21/16 10:22 PM, Joe Witt wrote:
>>>
>>>>
>>>> Jake,
>>>>
>>>> Can you give \\n a try on the regex replace strategy? Let us know if
>>>> that does it.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <ja...@yahoo.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> Is it possible to use ReplaceText processor to insert a newline
>>>>> character
>>>>> (0x0a) or other special characters into a flowfile? I've tried putting
>>>>> \r
>>>>> and \n in the value of ReplaceText's Replacement Value property, but
>>>>> they
>>>>> are inserted literally into the file (the Regex Replace strategy will
>>>>> insert
>>>>> an 'n', and Literal Replace strategy will insert '\n'). Is there
>>>>> another
>>>>> way to do it?
>>>>>
>>>>> Thanks,
>>>>> Jake
>>>>>
>>>>
Re: Using ReplaceText to insert newline character into flowfile?
Posted by Jake Kugel <ja...@yahoo.com>.
Thanks, creating the newline in the replacement value property with
shift-enter worked! The newline character (0x0a) was inserted into the
flowfile.
My use case is slightly more involved, I also need to insert carriage
returns (0x0d) in some cases. But I think you've given me a direction
to go, I'll see if I can copy/paste a carriage return character into the
replacement value.
Thanks again,
Jake
On 7/21/16 10:38 PM, Joe Witt wrote:
> Jake,
>
> In the 'Replacement Value' you can tell it what you want then you can
> hit "Shift-Enter" and it will insert the newline.
>
> So, lets say the input you're getting into this processor is a
> flowfile containing a bunch of line seperated events. If you wanted
> to insert a newline in between each line you could use a config for
> ReplaceText such as
>
> Search value: (^.*$)
> Replacement value: $1<hit shift enter and you'll see it put a newline>
> Replacement strategy: Regex Replace
> Evaluation Mode: Line by line
>
> With such a config lets say you passed in data that was
>
> some line 1
> some line 2
> some line 3
>
> Then after your processor you'd end up with
>
> some line 1
>
> some line 2
>
> some line 3
>
>
> Does that get you where you want to be?
>
> Thanks
> Joe
>
> On Thu, Jul 21, 2016 at 11:28 PM, Jake Kugel <ja...@yahoo.com> wrote:
>> Thanks for the reply. I tried just now using \\n with the regex replace
>> strategy, and it put '\n' into the flowfile instead of 'n'.
>>
>> Jake
>>
>>
>> On 7/21/16 10:22 PM, Joe Witt wrote:
>>>
>>> Jake,
>>>
>>> Can you give \\n a try on the regex replace strategy? Let us know if
>>> that does it.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <ja...@yahoo.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> Is it possible to use ReplaceText processor to insert a newline character
>>>> (0x0a) or other special characters into a flowfile? I've tried putting
>>>> \r
>>>> and \n in the value of ReplaceText's Replacement Value property, but they
>>>> are inserted literally into the file (the Regex Replace strategy will
>>>> insert
>>>> an 'n', and Literal Replace strategy will insert '\n'). Is there another
>>>> way to do it?
>>>>
>>>> Thanks,
>>>> Jake
Re: Using ReplaceText to insert newline character into flowfile?
Posted by Joe Witt <jo...@gmail.com>.
Jake,
In the 'Replacement Value' you can tell it what you want then you can
hit "Shift-Enter" and it will insert the newline.
So, lets say the input you're getting into this processor is a
flowfile containing a bunch of line seperated events. If you wanted
to insert a newline in between each line you could use a config for
ReplaceText such as
Search value: (^.*$)
Replacement value: $1<hit shift enter and you'll see it put a newline>
Replacement strategy: Regex Replace
Evaluation Mode: Line by line
With such a config lets say you passed in data that was
some line 1
some line 2
some line 3
Then after your processor you'd end up with
some line 1
some line 2
some line 3
Does that get you where you want to be?
Thanks
Joe
On Thu, Jul 21, 2016 at 11:28 PM, Jake Kugel <ja...@yahoo.com> wrote:
> Thanks for the reply. I tried just now using \\n with the regex replace
> strategy, and it put '\n' into the flowfile instead of 'n'.
>
> Jake
>
>
> On 7/21/16 10:22 PM, Joe Witt wrote:
>>
>> Jake,
>>
>> Can you give \\n a try on the regex replace strategy? Let us know if
>> that does it.
>>
>> Thanks
>> Joe
>>
>> On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <ja...@yahoo.com> wrote:
>>>
>>> Hello,
>>>
>>> Is it possible to use ReplaceText processor to insert a newline character
>>> (0x0a) or other special characters into a flowfile? I've tried putting
>>> \r
>>> and \n in the value of ReplaceText's Replacement Value property, but they
>>> are inserted literally into the file (the Regex Replace strategy will
>>> insert
>>> an 'n', and Literal Replace strategy will insert '\n'). Is there another
>>> way to do it?
>>>
>>> Thanks,
>>> Jake
Re: Using ReplaceText to insert newline character into flowfile?
Posted by Jake Kugel <ja...@yahoo.com>.
Thanks for the reply. I tried just now using \\n with the regex replace
strategy, and it put '\n' into the flowfile instead of 'n'.
Jake
On 7/21/16 10:22 PM, Joe Witt wrote:
> Jake,
>
> Can you give \\n a try on the regex replace strategy? Let us know if
> that does it.
>
> Thanks
> Joe
>
> On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <ja...@yahoo.com> wrote:
>> Hello,
>>
>> Is it possible to use ReplaceText processor to insert a newline character
>> (0x0a) or other special characters into a flowfile? I've tried putting \r
>> and \n in the value of ReplaceText's Replacement Value property, but they
>> are inserted literally into the file (the Regex Replace strategy will insert
>> an 'n', and Literal Replace strategy will insert '\n'). Is there another
>> way to do it?
>>
>> Thanks,
>> Jake
Re: Using ReplaceText to insert newline character into flowfile?
Posted by Joe Witt <jo...@gmail.com>.
Jake,
Can you give \\n a try on the regex replace strategy? Let us know if
that does it.
Thanks
Joe
On Thu, Jul 21, 2016 at 11:18 PM, Jake Kugel <ja...@yahoo.com> wrote:
> Hello,
>
> Is it possible to use ReplaceText processor to insert a newline character
> (0x0a) or other special characters into a flowfile? I've tried putting \r
> and \n in the value of ReplaceText's Replacement Value property, but they
> are inserted literally into the file (the Regex Replace strategy will insert
> an 'n', and Literal Replace strategy will insert '\n'). Is there another
> way to do it?
>
> Thanks,
> Jake