You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2021/04/20 00:15:45 UTC
ExecuteStreamCommand for piped commands
Good evening. I’d like to do a file count of ${directory}. I can use an
ExecuteStreamCommand processor with that as Command Arguments and ls as
Command Path. I’ll set Output Destination Attribute to be myFileCount, and
will drop Max Attribute Length to ten. That will be ample characters for my
counts, with plenty of margin.
How though, do I incorporate the “| wc -l” (double quotes for clarity only,
not part of the command) in the processor configuration so that I get the
desired count, not the actual listing of files? I need this count to
RouteOnAttribute as a next step.
Thanks very much for your help.
Re: ExecuteStreamCommand for piped commands
Posted by James McMahon <js...@gmail.com>.
Thanks very much. Will try it momentarily, this morning.
On Mon, Apr 19, 2021 at 9:30 PM Otto Fowler <ot...@gmail.com> wrote:
> Piping is part of the shell I believe, so you can’t do it with arguments
> in that way.
>
> If you make your shell the executable however, you should then be able to
> pass things as separate
> arguments.
>
> I would start with
>
> "/bin/sh” as the executable and arguments of
>
> "-c”
>
> “ls | ws -l”
>
>
> Or, you can write what you want to do in a .sh script file, and execute that.
>
>
>
> On Apr 19, 2021, at 20:15, James McMahon <js...@gmail.com> wrote:
>
> Good evening. I’d like to do a file count of ${directory}. I can use an
> ExecuteStreamCommand processor with that as Command Arguments and ls as
> Command Path. I’ll set Output Destination Attribute to be myFileCount, and
> will drop Max Attribute Length to ten. That will be ample characters for my
> counts, with plenty of margin.
>
> How though, do I incorporate the “| wc -l” (double quotes for clarity
> only, not part of the command) in the processor configuration so that I get
> the desired count, not the actual listing of files? I need this count to
> RouteOnAttribute as a next step.
>
> Thanks very much for your help.
>
>
>
Re: ExecuteStreamCommand for piped commands
Posted by Otto Fowler <ot...@gmail.com>.
I can’t imagine it taking that long to do that no, although environment may make a difference?
If you can come up with a reproduction, it might be worth creating a jira issue with a reproduction template or something
> On Apr 20, 2021, at 07:09, James McMahon <js...@gmail.com> wrote:
>
> I opted to call a bash script as Command Path. I felt the script approach gave me a lot more flexibility to do just about anything I want without worrying about syntax of arguments, quoting, etc etc in the ExecuteScriptCommand itself. If anyone is interested this is my simple example:
>
> ExecuteStreamCommand config:
> Command Arguments ... /abc/def/${myDirectory}
> Command Path ... ${scriptsBaseDir}/util/directoryCount.sh
> Ignore STDIN ... false
> Working Directory ... No value set
> Argument Delimiter ... ;
> Output Destination Attribute ... directoryCount
> Max Attribute Length ... 20
>
> My simple script:
>
> #!/bin/bash
> echo `ls $1 | wc -l`
> exit 0
>
> Again this script is silly simple, but the point here is the framework itself. You could work all sorts of magic and complex logic into the script that you please, which otherwise might be more challenging to set up in the ExecuteStreamCommand processor itself.
>
> One observation: it takes about 20-30 seconds for ExecuteStreamCommand to take my incoming flowfile and produce the output. Could this extensive time be due to a shell instance actually being created under the hood by ExecuteStreamCommand? I suspect I'm missing something about this.
>
> Otto, thanks very much once again for the good suggestions.
>
> On Mon, Apr 19, 2021 at 9:30 PM Otto Fowler <ottobackwards@gmail.com <ma...@gmail.com>> wrote:
> Piping is part of the shell I believe, so you can’t do it with arguments in that way.
>
> If you make your shell the executable however, you should then be able to pass things as separate
> arguments.
>
> I would start with
>
> "/bin/sh” as the executable and arguments of
> "-c”
> “ls | ws -l”
>
> Or, you can write what you want to do in a .sh script file, and execute that.
>
>
>> On Apr 19, 2021, at 20:15, James McMahon <jsmcmahon3@gmail.com <ma...@gmail.com>> wrote:
>>
>> Good evening. I’d like to do a file count of ${directory}. I can use an ExecuteStreamCommand processor with that as Command Arguments and ls as Command Path. I’ll set Output Destination Attribute to be myFileCount, and will drop Max Attribute Length to ten. That will be ample characters for my counts, with plenty of margin.
>>
>> How though, do I incorporate the “| wc -l” (double quotes for clarity only, not part of the command) in the processor configuration so that I get the desired count, not the actual listing of files? I need this count to RouteOnAttribute as a next step.
>>
>> Thanks very much for your help.
>
Re: ExecuteStreamCommand for piped commands
Posted by James McMahon <js...@gmail.com>.
I opted to call a bash script as Command Path. I felt the script approach
gave me a lot more flexibility to do just about anything I want without
worrying about syntax of arguments, quoting, etc etc in the
ExecuteScriptCommand itself. If anyone is interested this is my simple
example:
ExecuteStreamCommand config:
Command Arguments ... /abc/def/${myDirectory}
Command Path ... ${scriptsBaseDir}/util/directoryCount.sh
Ignore STDIN ... false
Working Directory ... No value set
Argument Delimiter ... ;
Output Destination Attribute ... directoryCount
Max Attribute Length ... 20
My simple script:
#!/bin/bash
echo `ls $1 | wc -l`
exit 0
Again this script is silly simple, but the point here is the framework
itself. You could work all sorts of magic and complex logic into the script
that you please, which otherwise might be more challenging to set up in the
ExecuteStreamCommand processor itself.
One observation: it takes about 20-30 seconds for ExecuteStreamCommand to
take my incoming flowfile and produce the output. Could this extensive time
be due to a shell instance actually being created under the hood by
ExecuteStreamCommand? I suspect I'm missing something about this.
Otto, thanks very much once again for the good suggestions.
On Mon, Apr 19, 2021 at 9:30 PM Otto Fowler <ot...@gmail.com> wrote:
> Piping is part of the shell I believe, so you can’t do it with arguments
> in that way.
>
> If you make your shell the executable however, you should then be able to
> pass things as separate
> arguments.
>
> I would start with
>
> "/bin/sh” as the executable and arguments of
>
> "-c”
>
> “ls | ws -l”
>
>
> Or, you can write what you want to do in a .sh script file, and execute that.
>
>
>
> On Apr 19, 2021, at 20:15, James McMahon <js...@gmail.com> wrote:
>
> Good evening. I’d like to do a file count of ${directory}. I can use an
> ExecuteStreamCommand processor with that as Command Arguments and ls as
> Command Path. I’ll set Output Destination Attribute to be myFileCount, and
> will drop Max Attribute Length to ten. That will be ample characters for my
> counts, with plenty of margin.
>
> How though, do I incorporate the “| wc -l” (double quotes for clarity
> only, not part of the command) in the processor configuration so that I get
> the desired count, not the actual listing of files? I need this count to
> RouteOnAttribute as a next step.
>
> Thanks very much for your help.
>
>
>
Re: ExecuteStreamCommand for piped commands
Posted by Otto Fowler <ot...@gmail.com>.
Piping is part of the shell I believe, so you can’t do it with arguments in that way.
If you make your shell the executable however, you should then be able to pass things as separate
arguments.
I would start with
"/bin/sh” as the executable and arguments of
"-c”
“ls | ws -l”
Or, you can write what you want to do in a .sh script file, and execute that.
> On Apr 19, 2021, at 20:15, James McMahon <js...@gmail.com> wrote:
>
> Good evening. I’d like to do a file count of ${directory}. I can use an ExecuteStreamCommand processor with that as Command Arguments and ls as Command Path. I’ll set Output Destination Attribute to be myFileCount, and will drop Max Attribute Length to ten. That will be ample characters for my counts, with plenty of margin.
>
> How though, do I incorporate the “| wc -l” (double quotes for clarity only, not part of the command) in the processor configuration so that I get the desired count, not the actual listing of files? I need this count to RouteOnAttribute as a next step.
>
> Thanks very much for your help.