You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by sanjeet rath <ra...@gmail.com> on 2020/04/10 15:22:46 UTC
Storing output of shellscript in s3 bucket
Hi,
I have a scenario , where i have to Execute a shell script amd the output
of the script is a json file and i want to put the file in s3 bucket.
I am able to do it by building 2 flows.
One flow for executestremproces and store the json file in folder in System
Then another flow to get the file from system and using puts3object to put
in s3 bucket. But building write and read in separate flow is bit risky and
noway i am make it dependent as getfile has no preprocessor.
Is it possible not to store the json file (output of shell script) in file
system and as a flow file move it to s3 bucket ?
Means in one flow to execute shellscript and store output in s3 bucket.
Regards,
Sanjeet
Re: Storing output of shellscript in s3 bucket
Posted by Andy LoPresto <al...@apache.org>.
If that script cannot write directly to STDOUT in this scenario (which the docs do not make clear), there are multiple ways to achieve what you need.
1. Instead of invoking the command directly in the ExecuteStreamCommand processor, invoke a custom script file “example_zk_script.sh zk_source_data.json” which contains:
```
zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/“$@“
cat /path/to/export/"$@"
```
This script will execute the command you were originally calling (which sends data _to_ ZooKeeper that is read _from_ the file location you provided) and then output the file contents to STDOUT, which will be captured as the flowfile content.
2. Read directly from the file using a FetchFile processor. The path to the file is known in the context of NiFi, either as a static value in the ExecuteStreamCommand property descriptor, or as an attribute or variable, so that file path can be read from directly using another processor. The FetchFile processor replaces a flowfile’s content with the content of a file on disk, so pass the result of the ExecuteStreamCommand to the FetchFile processor with the path in the filename attribute (update using an UpdateAttribute processor if necessary).
I would recommend the first option as a cleaner and more robust solution.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 10, 2020, at 7:19 PM, Joe Witt <jo...@gmail.com> wrote:
>
> If the zk-migrator can be configured to write to stdout instead of a file then yes.
>
> On Fri, Apr 10, 2020 at 3:52 PM sanjeet rath <rath.sanjeet@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> Thanks for your quick reply.Yeah i am using executestreamcommand to execute bellow script
>
> zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components -f /path/to/export/zk-source-data.json
>
> Can the zk-source-data.json file written as a output flow file of the above processor.if yes please let me know how.
>
> Many thanks
> sanjeet
>
> On Fri, 10 Apr 2020, 9:25 pm Bryan Bende, <bbende@gmail.com <ma...@gmail.com>> wrote:
> Hello,
>
> Assuming you are using the ExecuteStreamCommand processors, then the output of the command is written to the flow file content. So if your command writes the JSON to stdout, then it should end up in the flow file.
>
> Thanks,
>
> Bryan
>
>
> On Fri, Apr 10, 2020 at 11:23 AM sanjeet rath <rath.sanjeet@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I have a scenario , where i have to Execute a shell script amd the output of the script is a json file and i want to put the file in s3 bucket.
>
> I am able to do it by building 2 flows.
>
> One flow for executestremproces and store the json file in folder in System
>
> Then another flow to get the file from system and using puts3object to put in s3 bucket. But building write and read in separate flow is bit risky and noway i am make it dependent as getfile has no preprocessor.
>
>
> Is it possible not to store the json file (output of shell script) in file system and as a flow file move it to s3 bucket ?
> Means in one flow to execute shellscript and store output in s3 bucket.
>
> Regards,
> Sanjeet
>
Re: Storing output of shellscript in s3 bucket
Posted by Joe Witt <jo...@gmail.com>.
If the zk-migrator can be configured to write to stdout instead of a file
then yes.
On Fri, Apr 10, 2020 at 3:52 PM sanjeet rath <ra...@gmail.com> wrote:
> Hi,
>
> Thanks for your quick reply.Yeah i am using executestreamcommand to
> execute bellow script
>
> zk-migrator.sh -s -z destinationHostname:destinationClientPort/destinationRootPath/components
> -f /path/to/export/zk-source-data.json
>
> Can the zk-source-data.json file written as a output flow file of the
> above processor.if yes please let me know how.
>
> Many thanks
> sanjeet
>
> On Fri, 10 Apr 2020, 9:25 pm Bryan Bende, <bb...@gmail.com> wrote:
>
>> Hello,
>>
>> Assuming you are using the ExecuteStreamCommand processors, then the
>> output of the command is written to the flow file content. So if your
>> command writes the JSON to stdout, then it should end up in the flow file.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Fri, Apr 10, 2020 at 11:23 AM sanjeet rath <ra...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a scenario , where i have to Execute a shell script amd the
>>> output of the script is a json file and i want to put the file in s3 bucket.
>>>
>>> I am able to do it by building 2 flows.
>>>
>>> One flow for executestremproces and store the json file in folder in
>>> System
>>>
>>> Then another flow to get the file from system and using puts3object to
>>> put in s3 bucket. But building write and read in separate flow is bit risky
>>> and noway i am make it dependent as getfile has no preprocessor.
>>>
>>>
>>> Is it possible not to store the json file (output of shell script) in
>>> file system and as a flow file move it to s3 bucket ?
>>> Means in one flow to execute shellscript and store output in s3 bucket.
>>>
>>> Regards,
>>> Sanjeet
>>>
>>>
Re: Storing output of shellscript in s3 bucket
Posted by sanjeet rath <ra...@gmail.com>.
Hi,
Thanks for your quick reply.Yeah i am using executestreamcommand to execute
bellow script
zk-migrator.sh -s -z
destinationHostname:destinationClientPort/destinationRootPath/components
-f /path/to/export/zk-source-data.json
Can the zk-source-data.json file written as a output flow file of the above
processor.if yes please let me know how.
Many thanks
sanjeet
On Fri, 10 Apr 2020, 9:25 pm Bryan Bende, <bb...@gmail.com> wrote:
> Hello,
>
> Assuming you are using the ExecuteStreamCommand processors, then the
> output of the command is written to the flow file content. So if your
> command writes the JSON to stdout, then it should end up in the flow file.
>
> Thanks,
>
> Bryan
>
>
> On Fri, Apr 10, 2020 at 11:23 AM sanjeet rath <ra...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have a scenario , where i have to Execute a shell script amd the output
>> of the script is a json file and i want to put the file in s3 bucket.
>>
>> I am able to do it by building 2 flows.
>>
>> One flow for executestremproces and store the json file in folder in
>> System
>>
>> Then another flow to get the file from system and using puts3object to
>> put in s3 bucket. But building write and read in separate flow is bit risky
>> and noway i am make it dependent as getfile has no preprocessor.
>>
>>
>> Is it possible not to store the json file (output of shell script) in
>> file system and as a flow file move it to s3 bucket ?
>> Means in one flow to execute shellscript and store output in s3 bucket.
>>
>> Regards,
>> Sanjeet
>>
>>
Re: Storing output of shellscript in s3 bucket
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
Assuming you are using the ExecuteStreamCommand processors, then the output
of the command is written to the flow file content. So if your command
writes the JSON to stdout, then it should end up in the flow file.
Thanks,
Bryan
On Fri, Apr 10, 2020 at 11:23 AM sanjeet rath <ra...@gmail.com>
wrote:
> Hi,
>
> I have a scenario , where i have to Execute a shell script amd the output
> of the script is a json file and i want to put the file in s3 bucket.
>
> I am able to do it by building 2 flows.
>
> One flow for executestremproces and store the json file in folder in System
>
> Then another flow to get the file from system and using puts3object to put
> in s3 bucket. But building write and read in separate flow is bit risky and
> noway i am make it dependent as getfile has no preprocessor.
>
>
> Is it possible not to store the json file (output of shell script) in file
> system and as a flow file move it to s3 bucket ?
> Means in one flow to execute shellscript and store output in s3 bucket.
>
> Regards,
> Sanjeet
>
>