You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by jose antonio rodriguez diaz <jo...@gmail.com> on 2016/02/28 22:53:35 UTC
How to configure a ExecuteStreamCommand
Hello All,
I am just getting started with apache Nifi doing a kind of PoC (Proof of Concept) My DataFlow is compose as follow
GetFile->PutFile->ExecuteStreamCommand
The idea is move a file from a folder to another one and then execute an script. The first step (move the file from one side to the other) works perfectly but I haven´t been able to execute the script. The script is very simple I just want to create a file on my desktop.
the script called script.sh is located on my Desktop ($HOME/Desktop/script.sh)
#!/bin/bash
echo "This is a test" >> /Users/joseantoniorodriguez/Desktop/foo.txt
Also the ExecuteStreamCommand is configured as follow
Command Path: /Users/joseantoniorodriguez/Desktop/script.sh
Ignore STDIN: true
Working directory: /Users/joseantoniorodriguez/Desktop
Argument delimiter: ;
Output destination attribute: /Users/joseantoniorodriguez/Desktop —> ¿Is this necessary?
Max attribute length: /Users/joseantoniorodriguez/Desktop
The file I’m using to test are both csv about one of 324KB and the other 22MB.
After execute I could see the file has been moved from one folder to the other but I did´t see any foo.txt file on my desktop, also I did´t see any error on the Flow.
Could anybody give me a hand with this I am pretty sure this should be a ridiculous error or misconfiguration. BY the way the OS is Mac OS X.
Thanks in advance.
Re: How to configure a ExecuteStreamCommand
Posted by Joe Percivall <jo...@yahoo.com>.
Hello,
Glad you were able to get it working.
NiFi was designed to work just as well during failures as successes. If you notice, there are "success" and "failure" relationships coming out of the PutFile processor. You can create a connection for the failure relationship in the same way you did for success.
So if you purely just want to log failures then route the failure relationship to the LogAttribute processor. If you want more immediate notifications, you could also use the PutEmail processor. It all depends on what your system needs are.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Thursday, March 3, 2016 2:50 AM, jose antonio rodriguez diaz <jo...@gmail.com> wrote:
Hello All,
Finally I could make it work. In addition to errors identified by Joe, I had another errors, specifically on the connection between PutFile and ExecuteStreamProcess, success relation wasn’t checked, so to try to improve this data flow what could I do to log failure relationships, what do you suggest? Any comment will be nice to ear, no matter if it is about any other way to improve the whole data flow.
Thanks again Joe for answer me.
Regards
> El 3 mar 2016, a las 0:12, jose antonio rodriguez diaz <jo...@gmail.com> escribió:
>
> Hello Joe,
>
> I am gonna explain you the whole picture of what I got and what I would like to have. Right now we I receive a file (In fact I receive several files) in a shared net unit called “Z" then manually I move the file to a local folder called “data” and finally I execute a batch file, this batch file consume (read) the file and after read it move it to another folder called “imported”. That’s why I just try to invoke a batch program once the file has been dropped on the “data” folder.
>
> I have changed Max attribute length and set it to 256 and also leaved the Output destination attribute empty. Even so again I haven’t been able to execute the batch file (there is no new file called foo.txt on my desktop).
>
> Have you any idea what I am doing wrong? I am pretty sure should be an easy fix. Please fell free to make any comment or suggestion regarding to my case.
>
> Thanks in advance.
>
>> El 2 mar 2016, a las 17:36, Joe Percivall <jo...@yahoo.com> escribió:
>>
>> Hello,
>>
>> Welcome to NiFi!
>>
>> I just tried running an ExecuteStreamCommand processor with the properties you have (I created a script and modified the paths to point to a folder that exists) and two things jump out. One, the Max attribute length must take an integer. If you set it to be a path the processor will be invalid and you'll see a yellow warning icon in the top left of the processor. This means the processor will not run and you'll see the flowfiles queue up in the relationship preceding it.
>>
>> Second, the Output Destination Attribute is only for when you want to output the results of the command to an attribute instead of the content of a new flowfile (useful for running a command to find the character encoding of the contents). Using an integer for the max attribute length I am able to correctly run the script.
>>
>> As a helpful hint, you can see the description of a property by hovering over the light blue "?" icon in the configure processor tab. Also you can see the documentation for the processor by right clicking on it and selecting "usage" from the list.
>>
>> Also what will you eventually be doing with your script? The way the ExecuteStreamCommand is designed to work is by taking in a FlowFile and then running an external command on it. So you may make your flow more efficient and user friendly by putting the ExecuteStreamCommand between the Get and Put.
>>
>> Hope that helps,
>> Joe
>> - - - - - -
>> Joseph Percivall
>> linkedin.com/in/Percivall
>> e: joepercivall@yahoo.com
>>
>>
>>
>>
>> On Sunday, February 28, 2016 4:53 PM, jose antonio rodriguez diaz <jo...@gmail.com> wrote:
>> Hello All,
>>
>> I am just getting started with apache Nifi doing a kind of PoC (Proof of Concept) My DataFlow is compose as follow
>> GetFile->PutFile->ExecuteStreamCommand
>>
>> The idea is move a file from a folder to another one and then execute an script. The first step (move the file from one side to the other) works perfectly but I haven´t been able to execute the script. The script is very simple I just want to create a file on my desktop.
>>
>> the script called script.sh is located on my Desktop ($HOME/Desktop/script.sh)
>>
>> #!/bin/bash
>>
>> echo "This is a test" >> /Users/joseantoniorodriguez/Desktop/foo.txt
>>
>>
>>
>>
>> Also the ExecuteStreamCommand is configured as follow
>>
>> Command Path: /Users/joseantoniorodriguez/Desktop/script.sh
>> Ignore STDIN: true
>> Working directory: /Users/joseantoniorodriguez/Desktop
>> Argument delimiter: ;
>> Output destination attribute: /Users/joseantoniorodriguez/Desktop —> ¿Is this necessary?
>> Max attribute length: /Users/joseantoniorodriguez/Desktop
>>
>>
>> The file I’m using to test are both csv about one of 324KB and the other 22MB.
>>
>> After execute I could see the file has been moved from one folder to the other but I did´t see any foo.txt file on my desktop, also I did´t see any error on the Flow.
>>
>> Could anybody give me a hand with this I am pretty sure this should be a ridiculous error or misconfiguration. BY the way the OS is Mac OS X.
>>
>> Thanks in advance.
>
Re: How to configure a ExecuteStreamCommand
Posted by jose antonio rodriguez diaz <jo...@gmail.com>.
Hello All,
Finally I could make it work. In addition to errors identified by Joe, I had another errors, specifically on the connection between PutFile and ExecuteStreamProcess, success relation wasn’t checked, so to try to improve this data flow what could I do to log failure relationships, what do you suggest? Any comment will be nice to ear, no matter if it is about any other way to improve the whole data flow.
Thanks again Joe for answer me.
Regards
> El 3 mar 2016, a las 0:12, jose antonio rodriguez diaz <jo...@gmail.com> escribió:
>
> Hello Joe,
>
> I am gonna explain you the whole picture of what I got and what I would like to have. Right now we I receive a file (In fact I receive several files) in a shared net unit called “Z" then manually I move the file to a local folder called “data” and finally I execute a batch file, this batch file consume (read) the file and after read it move it to another folder called “imported”. That’s why I just try to invoke a batch program once the file has been dropped on the “data” folder.
>
> I have changed Max attribute length and set it to 256 and also leaved the Output destination attribute empty. Even so again I haven’t been able to execute the batch file (there is no new file called foo.txt on my desktop).
>
> Have you any idea what I am doing wrong? I am pretty sure should be an easy fix. Please fell free to make any comment or suggestion regarding to my case.
>
> Thanks in advance.
>
>> El 2 mar 2016, a las 17:36, Joe Percivall <jo...@yahoo.com> escribió:
>>
>> Hello,
>>
>> Welcome to NiFi!
>>
>> I just tried running an ExecuteStreamCommand processor with the properties you have (I created a script and modified the paths to point to a folder that exists) and two things jump out. One, the Max attribute length must take an integer. If you set it to be a path the processor will be invalid and you'll see a yellow warning icon in the top left of the processor. This means the processor will not run and you'll see the flowfiles queue up in the relationship preceding it.
>>
>> Second, the Output Destination Attribute is only for when you want to output the results of the command to an attribute instead of the content of a new flowfile (useful for running a command to find the character encoding of the contents). Using an integer for the max attribute length I am able to correctly run the script.
>>
>> As a helpful hint, you can see the description of a property by hovering over the light blue "?" icon in the configure processor tab. Also you can see the documentation for the processor by right clicking on it and selecting "usage" from the list.
>>
>> Also what will you eventually be doing with your script? The way the ExecuteStreamCommand is designed to work is by taking in a FlowFile and then running an external command on it. So you may make your flow more efficient and user friendly by putting the ExecuteStreamCommand between the Get and Put.
>>
>> Hope that helps,
>> Joe
>> - - - - - -
>> Joseph Percivall
>> linkedin.com/in/Percivall
>> e: joepercivall@yahoo.com
>>
>>
>>
>>
>> On Sunday, February 28, 2016 4:53 PM, jose antonio rodriguez diaz <jo...@gmail.com> wrote:
>> Hello All,
>>
>> I am just getting started with apache Nifi doing a kind of PoC (Proof of Concept) My DataFlow is compose as follow
>> GetFile->PutFile->ExecuteStreamCommand
>>
>> The idea is move a file from a folder to another one and then execute an script. The first step (move the file from one side to the other) works perfectly but I haven´t been able to execute the script. The script is very simple I just want to create a file on my desktop.
>>
>> the script called script.sh is located on my Desktop ($HOME/Desktop/script.sh)
>>
>> #!/bin/bash
>>
>> echo "This is a test" >> /Users/joseantoniorodriguez/Desktop/foo.txt
>>
>>
>>
>>
>> Also the ExecuteStreamCommand is configured as follow
>>
>> Command Path: /Users/joseantoniorodriguez/Desktop/script.sh
>> Ignore STDIN: true
>> Working directory: /Users/joseantoniorodriguez/Desktop
>> Argument delimiter: ;
>> Output destination attribute: /Users/joseantoniorodriguez/Desktop —> ¿Is this necessary?
>> Max attribute length: /Users/joseantoniorodriguez/Desktop
>>
>>
>> The file I’m using to test are both csv about one of 324KB and the other 22MB.
>>
>> After execute I could see the file has been moved from one folder to the other but I did´t see any foo.txt file on my desktop, also I did´t see any error on the Flow.
>>
>> Could anybody give me a hand with this I am pretty sure this should be a ridiculous error or misconfiguration. BY the way the OS is Mac OS X.
>>
>> Thanks in advance.
>
Re: How to configure a ExecuteStreamCommand
Posted by jose antonio rodriguez diaz <jo...@gmail.com>.
Hello Joe,
I am gonna explain you the whole picture of what I got and what I would like to have. Right now we I receive a file (In fact I receive several files) in a shared net unit called “Z" then manually I move the file to a local folder called “data” and finally I execute a batch file, this batch file consume (read) the file and after read it move it to another folder called “imported”. That’s why I just try to invoke a batch program once the file has been dropped on the “data” folder.
I have changed Max attribute length and set it to 256 and also leaved the Output destination attribute empty. Even so again I haven’t been able to execute the batch file (there is no new file called foo.txt on my desktop).
Have you any idea what I am doing wrong? I am pretty sure should be an easy fix. Please fell free to make any comment or suggestion regarding to my case.
Thanks in advance.
> El 2 mar 2016, a las 17:36, Joe Percivall <jo...@yahoo.com> escribió:
>
> Hello,
>
> Welcome to NiFi!
>
> I just tried running an ExecuteStreamCommand processor with the properties you have (I created a script and modified the paths to point to a folder that exists) and two things jump out. One, the Max attribute length must take an integer. If you set it to be a path the processor will be invalid and you'll see a yellow warning icon in the top left of the processor. This means the processor will not run and you'll see the flowfiles queue up in the relationship preceding it.
>
> Second, the Output Destination Attribute is only for when you want to output the results of the command to an attribute instead of the content of a new flowfile (useful for running a command to find the character encoding of the contents). Using an integer for the max attribute length I am able to correctly run the script.
>
> As a helpful hint, you can see the description of a property by hovering over the light blue "?" icon in the configure processor tab. Also you can see the documentation for the processor by right clicking on it and selecting "usage" from the list.
>
> Also what will you eventually be doing with your script? The way the ExecuteStreamCommand is designed to work is by taking in a FlowFile and then running an external command on it. So you may make your flow more efficient and user friendly by putting the ExecuteStreamCommand between the Get and Put.
>
> Hope that helps,
> Joe
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joepercivall@yahoo.com
>
>
>
>
> On Sunday, February 28, 2016 4:53 PM, jose antonio rodriguez diaz <jo...@gmail.com> wrote:
> Hello All,
>
> I am just getting started with apache Nifi doing a kind of PoC (Proof of Concept) My DataFlow is compose as follow
> GetFile->PutFile->ExecuteStreamCommand
>
> The idea is move a file from a folder to another one and then execute an script. The first step (move the file from one side to the other) works perfectly but I haven´t been able to execute the script. The script is very simple I just want to create a file on my desktop.
>
> the script called script.sh is located on my Desktop ($HOME/Desktop/script.sh)
>
> #!/bin/bash
>
> echo "This is a test" >> /Users/joseantoniorodriguez/Desktop/foo.txt
>
>
>
>
> Also the ExecuteStreamCommand is configured as follow
>
> Command Path: /Users/joseantoniorodriguez/Desktop/script.sh
> Ignore STDIN: true
> Working directory: /Users/joseantoniorodriguez/Desktop
> Argument delimiter: ;
> Output destination attribute: /Users/joseantoniorodriguez/Desktop —> ¿Is this necessary?
> Max attribute length: /Users/joseantoniorodriguez/Desktop
>
>
> The file I’m using to test are both csv about one of 324KB and the other 22MB.
>
> After execute I could see the file has been moved from one folder to the other but I did´t see any foo.txt file on my desktop, also I did´t see any error on the Flow.
>
> Could anybody give me a hand with this I am pretty sure this should be a ridiculous error or misconfiguration. BY the way the OS is Mac OS X.
>
> Thanks in advance.
Re: How to configure a ExecuteStreamCommand
Posted by Joe Percivall <jo...@yahoo.com>.
Hello,
Welcome to NiFi!
I just tried running an ExecuteStreamCommand processor with the properties you have (I created a script and modified the paths to point to a folder that exists) and two things jump out. One, the Max attribute length must take an integer. If you set it to be a path the processor will be invalid and you'll see a yellow warning icon in the top left of the processor. This means the processor will not run and you'll see the flowfiles queue up in the relationship preceding it.
Second, the Output Destination Attribute is only for when you want to output the results of the command to an attribute instead of the content of a new flowfile (useful for running a command to find the character encoding of the contents). Using an integer for the max attribute length I am able to correctly run the script.
As a helpful hint, you can see the description of a property by hovering over the light blue "?" icon in the configure processor tab. Also you can see the documentation for the processor by right clicking on it and selecting "usage" from the list.
Also what will you eventually be doing with your script? The way the ExecuteStreamCommand is designed to work is by taking in a FlowFile and then running an external command on it. So you may make your flow more efficient and user friendly by putting the ExecuteStreamCommand between the Get and Put.
Hope that helps,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Sunday, February 28, 2016 4:53 PM, jose antonio rodriguez diaz <jo...@gmail.com> wrote:
Hello All,
I am just getting started with apache Nifi doing a kind of PoC (Proof of Concept) My DataFlow is compose as follow
GetFile->PutFile->ExecuteStreamCommand
The idea is move a file from a folder to another one and then execute an script. The first step (move the file from one side to the other) works perfectly but I haven´t been able to execute the script. The script is very simple I just want to create a file on my desktop.
the script called script.sh is located on my Desktop ($HOME/Desktop/script.sh)
#!/bin/bash
echo "This is a test" >> /Users/joseantoniorodriguez/Desktop/foo.txt
Also the ExecuteStreamCommand is configured as follow
Command Path: /Users/joseantoniorodriguez/Desktop/script.sh
Ignore STDIN: true
Working directory: /Users/joseantoniorodriguez/Desktop
Argument delimiter: ;
Output destination attribute: /Users/joseantoniorodriguez/Desktop —> ¿Is this necessary?
Max attribute length: /Users/joseantoniorodriguez/Desktop
The file I’m using to test are both csv about one of 324KB and the other 22MB.
After execute I could see the file has been moved from one folder to the other but I did´t see any foo.txt file on my desktop, also I did´t see any error on the Flow.
Could anybody give me a hand with this I am pretty sure this should be a ridiculous error or misconfiguration. BY the way the OS is Mac OS X.
Thanks in advance.