You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Rob Weiss <ro...@g2-inc.com> on 2015/04/28 16:31:37 UTC
Running Python Scripts from NiFi
Any thoughts on how to accomplish this?
Re: Running Python Scripts from NiFi
Posted by Mark Payne <ma...@hotmail.com>.
Rob,
The processor has 2 relationships: "original" where the original
flowfile goes and "output stream" where a new FlowFile is routed. The
contents of this new flowfile are the output from the command's stdout.
Thanks
-Mark
------ Original Message ------
From: "Rob Weiss" <ro...@g2-inc.com>
To: dev@nifi.incubator.apache.org
Sent: 4/28/2015 10:54:11 AM
Subject: Re: Running Python Scripts from NiFi
>Mark,
>Thx for the heads up; we are trying not to touch disk during this
>process.
>Can I assume that STDOUT would be where we send the modified data to
>get it
>back into NiFi?
>
>Thanks Again!
>
>On Tue, Apr 28, 2015 at 10:51 AM, Mark Payne <ma...@hotmail.com>
>wrote:
>
>> Rob,
>>
>> The content of the FlowFile would be streamed to the Python's STDIN.
>>
>> So the Python script would have to be written to read from std in. If
>>it
>> instead is intended to run against a file, you could use PutFile to
>>write
>> the contents to some temporary directory.
>>
>> Thanks
>> -Mark
>>
>> ------ Original Message ------
>> From: "Rob Weiss" <ro...@g2-inc.com>
>> To: dev@nifi.incubator.apache.org
>> Sent: 4/28/2015 10:43:54 AM
>> Subject: Re: Running Python Scripts from NiFi
>>
>> Mark,
>>> How would that processor get access to the data in the flowfile?
>>>
>>> On Tue, Apr 28, 2015 at 10:40 AM, Mark Payne <ma...@hotmail.com>
>>> wrote:
>>>
>>> Rob,
>>>>
>>>> We used to have a processor for running scripts directly in NiFi.
>>>> However,
>>>> we have to remove that due to licensing conflicts with the
>>>>libraries
>>>> that
>>>> we were trying to use - they were not Apache friendly,
>>>>unfortunately.
>>>>
>>>> You could use the ExecuteStreamCommand processor in order to
>>>>accomplish
>>>> this though. It invokes whatever OS command you give it, so you
>>>>can use
>>>> it
>>>> to run a python script.
>>>>
>>>> Does that do what you need?
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>>
>>>> ------ Original Message ------
>>>> From: "Rob Weiss" <ro...@g2-inc.com>
>>>> To: dev@nifi.incubator.apache.org
>>>> Sent: 4/28/2015 10:31:37 AM
>>>> Subject: Running Python Scripts from NiFi
>>>>
>>>> Any thoughts on how to accomplish this?
>>>>
>>>>>
>>>>>
>>>>
Re: Running Python Scripts from NiFi
Posted by Rob Weiss <ro...@g2-inc.com>.
Mark,
Thx for the heads up; we are trying not to touch disk during this process.
Can I assume that STDOUT would be where we send the modified data to get it
back into NiFi?
Thanks Again!
On Tue, Apr 28, 2015 at 10:51 AM, Mark Payne <ma...@hotmail.com> wrote:
> Rob,
>
> The content of the FlowFile would be streamed to the Python's STDIN.
>
> So the Python script would have to be written to read from std in. If it
> instead is intended to run against a file, you could use PutFile to write
> the contents to some temporary directory.
>
> Thanks
> -Mark
>
> ------ Original Message ------
> From: "Rob Weiss" <ro...@g2-inc.com>
> To: dev@nifi.incubator.apache.org
> Sent: 4/28/2015 10:43:54 AM
> Subject: Re: Running Python Scripts from NiFi
>
> Mark,
>> How would that processor get access to the data in the flowfile?
>>
>> On Tue, Apr 28, 2015 at 10:40 AM, Mark Payne <ma...@hotmail.com>
>> wrote:
>>
>> Rob,
>>>
>>> We used to have a processor for running scripts directly in NiFi.
>>> However,
>>> we have to remove that due to licensing conflicts with the libraries
>>> that
>>> we were trying to use - they were not Apache friendly, unfortunately.
>>>
>>> You could use the ExecuteStreamCommand processor in order to accomplish
>>> this though. It invokes whatever OS command you give it, so you can use
>>> it
>>> to run a python script.
>>>
>>> Does that do what you need?
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> ------ Original Message ------
>>> From: "Rob Weiss" <ro...@g2-inc.com>
>>> To: dev@nifi.incubator.apache.org
>>> Sent: 4/28/2015 10:31:37 AM
>>> Subject: Running Python Scripts from NiFi
>>>
>>> Any thoughts on how to accomplish this?
>>>
>>>>
>>>>
>>>
Re: Running Python Scripts from NiFi
Posted by Mark Payne <ma...@hotmail.com>.
Rob,
The content of the FlowFile would be streamed to the Python's STDIN.
So the Python script would have to be written to read from std in. If it
instead is intended to run against a file, you could use PutFile to
write the contents to some temporary directory.
Thanks
-Mark
------ Original Message ------
From: "Rob Weiss" <ro...@g2-inc.com>
To: dev@nifi.incubator.apache.org
Sent: 4/28/2015 10:43:54 AM
Subject: Re: Running Python Scripts from NiFi
>Mark,
>How would that processor get access to the data in the flowfile?
>
>On Tue, Apr 28, 2015 at 10:40 AM, Mark Payne <ma...@hotmail.com>
>wrote:
>
>> Rob,
>>
>> We used to have a processor for running scripts directly in NiFi.
>>However,
>> we have to remove that due to licensing conflicts with the libraries
>>that
>> we were trying to use - they were not Apache friendly, unfortunately.
>>
>> You could use the ExecuteStreamCommand processor in order to
>>accomplish
>> this though. It invokes whatever OS command you give it, so you can
>>use it
>> to run a python script.
>>
>> Does that do what you need?
>>
>> Thanks
>> -Mark
>>
>>
>> ------ Original Message ------
>> From: "Rob Weiss" <ro...@g2-inc.com>
>> To: dev@nifi.incubator.apache.org
>> Sent: 4/28/2015 10:31:37 AM
>> Subject: Running Python Scripts from NiFi
>>
>> Any thoughts on how to accomplish this?
>>>
>>
Re: Running Python Scripts from NiFi
Posted by Rob Weiss <ro...@g2-inc.com>.
Mark,
How would that processor get access to the data in the flowfile?
On Tue, Apr 28, 2015 at 10:40 AM, Mark Payne <ma...@hotmail.com> wrote:
> Rob,
>
> We used to have a processor for running scripts directly in NiFi. However,
> we have to remove that due to licensing conflicts with the libraries that
> we were trying to use - they were not Apache friendly, unfortunately.
>
> You could use the ExecuteStreamCommand processor in order to accomplish
> this though. It invokes whatever OS command you give it, so you can use it
> to run a python script.
>
> Does that do what you need?
>
> Thanks
> -Mark
>
>
> ------ Original Message ------
> From: "Rob Weiss" <ro...@g2-inc.com>
> To: dev@nifi.incubator.apache.org
> Sent: 4/28/2015 10:31:37 AM
> Subject: Running Python Scripts from NiFi
>
> Any thoughts on how to accomplish this?
>>
>
Re: Running Python Scripts from NiFi
Posted by Mark Payne <ma...@hotmail.com>.
Rob,
We used to have a processor for running scripts directly in NiFi.
However, we have to remove that due to licensing conflicts with the
libraries that we were trying to use - they were not Apache friendly,
unfortunately.
You could use the ExecuteStreamCommand processor in order to accomplish
this though. It invokes whatever OS command you give it, so you can use
it to run a python script.
Does that do what you need?
Thanks
-Mark
------ Original Message ------
From: "Rob Weiss" <ro...@g2-inc.com>
To: dev@nifi.incubator.apache.org
Sent: 4/28/2015 10:31:37 AM
Subject: Running Python Scripts from NiFi
>Any thoughts on how to accomplish this?