You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Peter Wicks (pwicks)" <pw...@micron.com> on 2016/09/28 04:18:17 UTC
Remove top N lines from a text file
I have a CSV file where the first few lines are a summary of the report parameters that were used to generate it. I want to strip these off in NiFi.
I've considered using a RegEx to match the {N} top lines, but am wondering of a Groovy script might be a better option? I want to keep the file intact, so splitting it by line ending and routing all of the lines through a RouteByAttribute seems excessive.
I've never built a Groovy script, any examples on how I might go about this?
Thanks,
Peter
RE: Remove top N lines from a text file
Posted by "Carlos Manuel Fernandes (DSI)" <ca...@cgd.pt>.
Hi Peter , the simplest way I see, need to transverse the all file, using a processor property. Ex: N_lines (number of Lines to Strip on file)
import java.nio.charset.StandardCharsets
//Read the flowFile
def flowFile = session.get()
if (!flowFile) return
//Read Processor property
def nLinesToStrip=N_lines.value.toInteger() // Number of Lines to Strip
//Read flowFile and Write a new one without the first nLinesToStrip
def counter=0
try {
flowFile = session.write(flowFile, {inputStream, outputStream ->
inputStream.eachLine { line ->
counter++
if (counter > nLinesToStrip) {
outputStream.write("${line}\n".getBytes(StandardCharsets.UTF_8))
}
}
} as StreamCallback)
session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e) {
log.error(e)
session.transfer(flowFile, REL_FAILURE)
}
For better understanding how to build ExecuteScripts, in Groovy I recommend the blog : funnifi.blogspot.com from Matt Burgess<https://plus.google.com/u/0/116659654929723949312?prsrc=4>. Thanks Matt.
Carlos
From: Pierre Villard [mailto:pierre.villard.fr@gmail.com]
Sent: quarta-feira, 28 de Setembro de 2016 12:04
To: users@nifi.apache.org
Subject: Re: Remove top N lines from a text file
Hi Peter,
I would recommend you the following blog by Matt:
http://funnifi.blogspot.fr/2016/02/executescript-processor-replacing-flow.html
Pierre
2016-09-28 13:01 GMT+02:00 Andrew Grande <ap...@gmail.com>>:
Groovy script or a simple sed command invoked via ExecuteStreamingCommand should do the job.
Andrew
On Wed, Sep 28, 2016, 12:18 AM Peter Wicks (pwicks) <pw...@micron.com>> wrote:
I have a CSV file where the first few lines are a summary of the report parameters that were used to generate it. I want to strip these off in NiFi.
I’ve considered using a RegEx to match the {N} top lines, but am wondering of a Groovy script might be a better option? I want to keep the file intact, so splitting it by line ending and routing all of the lines through a RouteByAttribute seems excessive.
I’ve never built a Groovy script, any examples on how I might go about this?
Thanks,
Peter
Re: Remove top N lines from a text file
Posted by Mark Payne <ma...@hotmail.com>.
Peter,
Another option that may be a lot easier for you is to use the RouteText processor.
If you set Matching Strategy to "Satisfies Expression", you can use the Expression Language
to inspect FlowFile attributes, etc. But the RouteText processor also exposes two additional
variables: _line_ and _lineNo_ that you can use to route individual lines. So you set the Routing
Strategy to "Route to 'matched' if line matches all conditions" and add a
property with a value of: ${lineNo:gt(5)}
That would route the first 5 lines to to the 'unmatched' relationship (lineNo is 1 for the first line),
and all the other lines would be routed to the 'matched' relationship. Then you can just auto-terminate
'unmatched'.
This should help to avoid having to write any custom scripts.
Thanks
-Mark
> On Sep 28, 2016, at 7:03 AM, Pierre Villard <pi...@gmail.com> wrote:
>
> Hi Peter,
>
> I would recommend you the following blog by Matt:
> http://funnifi.blogspot.fr/2016/02/executescript-processor-replacing-flow.html <http://funnifi.blogspot.fr/2016/02/executescript-processor-replacing-flow.html>
>
> Pierre
>
> 2016-09-28 13:01 GMT+02:00 Andrew Grande <aperepel@gmail.com <ma...@gmail.com>>:
> Groovy script or a simple sed command invoked via ExecuteStreamingCommand should do the job.
>
> Andrew
>
>
> On Wed, Sep 28, 2016, 12:18 AM Peter Wicks (pwicks) <pwicks@micron.com <ma...@micron.com>> wrote:
> I have a CSV file where the first few lines are a summary of the report parameters that were used to generate it. I want to strip these off in NiFi.
>
> I’ve considered using a RegEx to match the {N} top lines, but am wondering of a Groovy script might be a better option? I want to keep the file intact, so splitting it by line ending and routing all of the lines through a RouteByAttribute seems excessive.
>
>
>
> I’ve never built a Groovy script, any examples on how I might go about this?
>
>
>
> Thanks,
>
> Peter
>
>
Re: Remove top N lines from a text file
Posted by Pierre Villard <pi...@gmail.com>.
Hi Peter,
I would recommend you the following blog by Matt:
http://funnifi.blogspot.fr/2016/02/executescript-processor-replacing-flow.html
Pierre
2016-09-28 13:01 GMT+02:00 Andrew Grande <ap...@gmail.com>:
> Groovy script or a simple sed command invoked via ExecuteStreamingCommand
> should do the job.
>
> Andrew
>
> On Wed, Sep 28, 2016, 12:18 AM Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
>> I have a CSV file where the first few lines are a summary of the report
>> parameters that were used to generate it. I want to strip these off in NiFi.
>>
>> I’ve considered using a RegEx to match the {N} top lines, but am
>> wondering of a Groovy script might be a better option? I want to keep the
>> file intact, so splitting it by line ending and routing all of the lines
>> through a RouteByAttribute seems excessive.
>>
>>
>>
>> I’ve never built a Groovy script, any examples on how I might go about
>> this?
>>
>>
>>
>> Thanks,
>>
>> Peter
>>
>
Re: Remove top N lines from a text file
Posted by Andrew Grande <ap...@gmail.com>.
Groovy script or a simple sed command invoked via ExecuteStreamingCommand
should do the job.
Andrew
On Wed, Sep 28, 2016, 12:18 AM Peter Wicks (pwicks) <pw...@micron.com>
wrote:
> I have a CSV file where the first few lines are a summary of the report
> parameters that were used to generate it. I want to strip these off in NiFi.
>
> I’ve considered using a RegEx to match the {N} top lines, but am wondering
> of a Groovy script might be a better option? I want to keep the file
> intact, so splitting it by line ending and routing all of the lines through
> a RouteByAttribute seems excessive.
>
>
>
> I’ve never built a Groovy script, any examples on how I might go about
> this?
>
>
>
> Thanks,
>
> Peter
>