You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nifi.apache.org by wizz <ro...@johansyah.com> on 2015/09/21 14:22:43 UTC

Dynamic Properties for Processor

Hi Guys,

I am new to Nifi, it looks great but I am exploring what it can be done. Is
it possible to have something similar to this flow: GetFile --> SplitText
--> GetHttp --> Zip?

1. GetFile gets file (dog.txt) which contains list of link:
http://pic.pedigreedatabase.com/dogbreeds/samoyed.jpg
http://pic.pedigreedatabase.com/dogbreeds/bichon_frise.jpg
http://pic.pedigreedatabase.com/dogbreeds/norwich_terrier.jpg
http://pic.pedigreedatabase.com/dogbreeds/west_highland_white_terrier.jpg

2. SplitText will get each of the line of text
3. The line feeds to GetHttp to property URL which it will download the file
and save it accordingly

Is this possible guys? It seems the properties of processor is immutable and
the only dynamic is when the expression language is allowed in the property.

Another question would be, the source text (dog.txt) it seems being executed
multiple times, how do I mark it that the NiFi got the inputs each line and
no repeat?

Thanks!!



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic Properties for Processor

Posted by wizz <ro...@johansyah.com>.

Hi Adrian and Joe,

Thanks for the useful links and suggestions. Definitely I will look around.
I guess that is it for now. Thanks once again guys.



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p2976.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic Properties for Processor

Posted by Andy LoPresto <al...@apache.org>.

Yanusha,

I’m sorry, I don’t quite understand your question.

If you mean, “is there a way to programmatically set processor properties”, yes. You can use the NiFi REST API to do this [1]. Click “Processors” then “PUT /processors/{id}”.

If you mean, “is there a way to add custom properties to a new processor”, yes. The developer guide explains how to build custom properties [2].

Did either of these answer your question?

[1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
[2] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#documenting-properties

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 5, 2016, at 10:09 AM, yanusha56 <a....@gmail.com> wrote:
> 
> Team,
> 
> 
> Do we have any custom processor built for the dynamic properties setting?
> 
> I am looking for the process which can set the values to respective
> attributes and also properties for each processor in data flow.
> 
> Could you please suggest the available approaches for the for doing the
> above.
> 
> Thanks in advance !!!
> 
> Yanusha
> 
> 
> 
> 
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p13534.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic Properties for Processor

Posted by yanusha56 <a....@gmail.com>.

Team,


Do we have any custom processor built for the dynamic properties setting?

I am looking for the process which can set the values to respective
attributes and also properties for each processor in data flow.

Could you please suggest the available approaches for the for doing the
above.

Thanks in advance !!!

Yanusha




--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p13534.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic Properties for Processor

Posted by Aldrin Piri <al...@gmail.com>.

In terms of why there are restrictions, it is about intended usage and how
given exposed properties are .  Dynamic properties on those processors that
have certain resources they are relying upon may not be as friendly as the
framework is to these dynamic changes.  Accordingly, the power of providing
the capability to handle these changes is squarely on the developer of the
component to determine and must be explicitly provided. Those properties
that do support EL have this information conveyed in their associated help
tooltip in the properties configuration as well as in the accompanying
Usage documentation.

Certainly understand and appreciate the needs that some users have, but
feel that this may map better to the Variable Registry feature proposal
that is being considered [1].  We would appreciate any additional ideas or
input in guiding that process.

In terms of contributing new processors, we have developed a Contributor's
Guide [2] identifying the mechanics of the process.  Our Developer Guide
[3] is a nice way to get insights into the API and framework as well as
typical extension patterns.  This should get you on your way for developing
components and offering up the code for incorporation into the codebase.
Of course, this developer list is also a great mechanism to help facilitate
that process for anything that may be unclear or questions you may have
along the way.  We look forward to any contributions you would like to
share with us!

Thanks!

[1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
[2] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide
[3] https://nifi.apache.org/developer-guide.html

On Thu, Sep 24, 2015 at 11:10 AM, wizz <ro...@johansyah.com> wrote:

> Hi Adrian,
>
> This gets very interesting! Firstly, thanks for the quick and immediate
> response. What you just mentioned hit the right spot.
>
> A few things, when you said "GetTwitter does not support dynamic properties
> for the credentials", it is true for GetTwitter and so it does for the
> properties in other processors currently in the NiFi. So I wonder, why
> there
> is such restrictions on updating all of the properties dynamically? Why
> don't NiFi core developers enable all properties to be dynamically while
> there is functionality on the expression language?
>
> For the PutFile, I will do as what you suggested. It is something that I am
> looking for but it just strange to me that not all properties support on
> the
> expression language :)
>
> Lastly, could you please direct me for contributing some new processor for
> the NiFi? No immediate promise but I would like to add Spring Social into
> NiFi.
>
> Thanks!!
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p2960.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Dynamic Properties for Processor

Posted by wizz <ro...@johansyah.com>.

Hi Adrian,

This gets very interesting! Firstly, thanks for the quick and immediate
response. What you just mentioned hit the right spot.

A few things, when you said "GetTwitter does not support dynamic properties
for the credentials", it is true for GetTwitter and so it does for the
properties in other processors currently in the NiFi. So I wonder, why there
is such restrictions on updating all of the properties dynamically? Why
don't NiFi core developers enable all properties to be dynamically while
there is functionality on the expression language?

For the PutFile, I will do as what you suggested. It is something that I am
looking for but it just strange to me that not all properties support on the
expression language :)

Lastly, could you please direct me for contributing some new processor for
the NiFi? No immediate promise but I would like to add Spring Social into
NiFi.

Thanks!!




--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p2960.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic Properties for Processor

Posted by Aldrin Piri <al...@gmail.com>.

You certainly are on the right path and have the right ideas, but there are
some details to work out.  The biggest item here is that GetTwitter does
not support dynamic properties for the credentials, largely because it does
not take FlowFiles in as input to the processor.  GetTwitter falls into the
class of "source" processors. Ultimately, I'd imagine this was due to how
the Twitter client library interacts with the service and keeping that
connection managed for continuously streaming data.  As a result, the
GetTwitter processor would have to be configured with the appropriate
credentials by hand as it is not currently possible to configure and
connect dynamically.

Some other points to clarify issues you are seeing:

Given EvaluateJsonPath's contract, it is interpreting your Directory
dynamic property you specified in its configuration as a JsonPath
expression, which in this case is invalid given the value you supplied.
You have the right idea to set that Directory property for use in your
PutFile, but that particular property would be better suited with an
UpdateAttribute that makes use of a value that was extracted by
EvaluateJsonPath.  For instance, your directory property in the
PutAttribute could use the "User Id" attribute via NiFi's Expression
Language [1].  Your directory property for your PutFile would then be
/Feeds/${User Id} or something similar depending on where on the filesystem
you wish for it to be rooted.

Let us know if this generates additional questions.

--aldrin

[1]
http://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html

On Thu, Sep 24, 2015 at 7:52 AM, wizz <ro...@johansyah.com> wrote:

> Hi Aldrin and Joe,
>
> Thanks for the clarification. I think the best is to show by picture. I
> made
> NiFi flow graph, please find it in this screenshot (1.png).
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n2955/1.png>
>
> It is not working one but it will do to explain what I am trying to
> achieve.
>
> The goal that I would like is to get JSON values from JSON, it contains the
> Twitter credentials, get the latest status updates of an user and save them
> to file based on the user ID.
>
> Here is the JSON format:
> {
>     "user_id": 123,
>     "key": "ABC",
>     "secret": "XYZ",
>     "token": "AAA",
>     "tokensecret: "BBB",
>     "twitter_userid": "1234"
> }
>
> I attached what is the settings for the EvaluateJsonPath, (2.png)
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n2955/2.png>
>
> And also, the error that I get which the directory property is missing,
> which I am hoping that the property can be set from the EvaluateJsonPath.
> (3.png)
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n2955/3.png>
>
> And the last processor is the PutFile (4.png)
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n2955/4.png>
>
> I am sure I did something wrong here in terms of the NiFi concepts and
> goals. This is one of the example, I would like to use NiFi for. By having
> dynamic values from external data (such as JSON), dictate what the NiFi
> should be doing (giving the key, token for Twitter processor for a example)
> and where to put the file (based on the User ID from JSON)
>
> It would be amazing and appreciated if you guys could direct me and suggest
> me.
>
> Thanks!!
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p2955.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Dynamic Properties for Processor

Posted by wizz <ro...@johansyah.com>.

Hi Aldrin and Joe,

Thanks for the clarification. I think the best is to show by picture. I made
NiFi flow graph, please find it in this screenshot (1.png).
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n2955/1.png>

It is not working one but it will do to explain what I am trying to achieve.

The goal that I would like is to get JSON values from JSON, it contains the
Twitter credentials, get the latest status updates of an user and save them
to file based on the user ID.

Here is the JSON format:
{
"user_id": 123,
"key": "ABC",
"secret": "XYZ",
"token": "AAA",
"tokensecret: "BBB",
"twitter_userid": "1234"
}

I attached what is the settings for the EvaluateJsonPath, (2.png)
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n2955/2.png>

And also, the error that I get which the directory property is missing,
which I am hoping that the property can be set from the EvaluateJsonPath.
(3.png)
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n2955/3.png>

And the last processor is the PutFile (4.png)
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n2955/4.png>

I am sure I did something wrong here in terms of the NiFi concepts and
goals. This is one of the example, I would like to use NiFi for. By having
dynamic values from external data (such as JSON), dictate what the NiFi
should be doing (giving the key, token for Twitter processor for a example)
and where to put the file (based on the User ID from JSON)

It would be amazing and appreciated if you guys could direct me and suggest
me.

Thanks!!

--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p2955.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic Properties for Processor

Posted by Aldrin Piri <al...@gmail.com>.

When you mention getting the value set for properties, are you referring to
values of the JSON payload you are receiving?  If so, check out
EvaluateJsonPath [1].  This would allow you to extract values to attributes
that could later be used throughout for dictating the context of the flow.
If not, could you please provide some additional information.

Regarding where the data producer "resides," either option works.  However,
we tend to do as much as we can within the flow so we can leverage all the
features that NiFi brings to the table.  This typically works out to
managing the items in flow we have access to (in terms of connectivity,
credentials, etc) and bringing them into the flow from sources where we do
not.  For your case of Twitter where we have functionality already present,
we would certainly make use of that.

Let us know if you have additional questions.

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.EvaluateJsonPath/index.html

On Tue, Sep 22, 2015 at 3:02 AM, wizz <ro...@johansyah.com> wrote:

> Hi Joe,
>
> Thanks for the input! It makes sense. Gonna try the invokeHTTP. Is there
> any
> way to put dynamic value for properties that is "Supports Expression
> Language: false"? Or is it possible to set values dynamically for many
> processors from the start of the flow?
>
> Why I need the dynamic value  properties? Because lets say I would like to
> do any value set for properties based on the data from GetFile from JSON
> and
> from the values it determine the context of the flow:
> {
>   "user_id": "123"
>   "url": "http://blah",
>   "dir": "blah",
>   "db_name": "blah"
> }
>
> And also, from your expertise, it is better to have the data producer
> inside
> NiFi such as GetTwitter or you make them outside NiFi and get it using
> GetKafka or GetFile. Btw, do you have any suggestion of framework to
> produce
> data outside NiFi?
>
> Many thanks for the speedy response!!
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p2908.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Dynamic Properties for Processor

Posted by wizz <ro...@johansyah.com>.

Hi Joe,

Thanks for the input! It makes sense. Gonna try the invokeHTTP. Is there any
way to put dynamic value for properties that is "Supports Expression
Language: false"? Or is it possible to set values dynamically for many
processors from the start of the flow?

Why I need the dynamic value  properties? Because lets say I would like to
do any value set for properties based on the data from GetFile from JSON and
from the values it determine the context of the flow:
{
  "user_id": "123"
  "url": "http://blah",
  "dir": "blah",
  "db_name": "blah"
}

And also, from your expertise, it is better to have the data producer inside
NiFi such as GetTwitter or you make them outside NiFi and get it using
GetKafka or GetFile. Btw, do you have any suggestion of framework to produce
data outside NiFi?

Many thanks for the speedy response!!



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895p2908.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic Properties for Processor

Posted by Joe Percivall <jo...@yahoo.com.INVALID>.

Here's a better formatted version. Email decided to switch back to RTF.

Welcome to NiFi!

For the second part, under the configuration for getFile if you uncheck "Keep Source File" it'll only get the dog.txt once then get rid of the file afterwards.

Also as a note, I am in the process of making a RouteText processor that will make it much easier to work with individual lines of text. You can check out the ticket here [3].

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.InvokeHTTP/index.html
[2] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
[3] https://issues.apache.org/jira/browse/NIFI-942

Hope this helps!
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com

- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com

On Monday, September 21, 2015 5:01 PM, Joe Percivall <jo...@yahoo.com.INVALID> wrote:

Welcome to NiFi!

This can certainly be done. The processor you're looking for to execute arbitrary URLs is the InvokeHTTP processor [1]. So your flow would be GetFile -> SplitText -> ExtractText (to get the URL to an attribute) -> InvokeHTTP. In the InvokeHTTP you'll use the expression language [2] to reference the proper attribute.
For the second part, under the configuration for getFile if you uncheck "Keep Source File" it'll only get the dog.txt once then get rid of the file afterwards.
Also as a note, I am in the process of making a RouteText processor that will make it much easier to work with individual lines of text. You can check out the ticket here [3].
[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.InvokeHTTP/index.html[2] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html[3] https://issues.apache.org/jira/browse/NIFI-942
Hope this helps!Joe- - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joepercivall@yahoo.com

On Monday, September 21, 2015 8:59 AM, wizz <ro...@johansyah.com> wrote:

Hi Guys,

I am new to Nifi, it looks great but I am exploring what it can be done. Is
it possible to have something similar to this flow: GetFile --> SplitText
--> GetHttp --> Zip?

1. GetFile gets file (dog.txt) which contains list of link:
http://pic.pedigreedatabase.com/dogbreeds/samoyed.jpg
http://pic.pedigreedatabase.com/dogbreeds/bichon_frise.jpg
http://pic.pedigreedatabase.com/dogbreeds/norwich_terrier.jpg
http://pic.pedigreedatabase.com/dogbreeds/west_highland_white_terrier.jpg

2. SplitText will get each of the line of text
3. The line feeds to GetHttp to property URL which it will download the file
and save it accordingly

Is this possible guys? It seems the properties of processor is immutable and
the only dynamic is when the expression language is allowed in the property.

Another question would be, the source text (dog.txt) it seems being executed
multiple times, how do I mark it that the NiFi got the inputs each line and
no repeat?

Thanks!!

--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-Properties-for-Processor-tp2895.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic Properties for Processor

Posted by Joe Percivall <jo...@yahoo.com.INVALID>.

Welcome to NiFi!

On Monday, September 21, 2015 8:59 AM, wizz <ro...@johansyah.com> wrote:

Hi Guys,

I am new to Nifi, it looks great but I am exploring what it can be done. Is
it possible to have something similar to this flow: GetFile --> SplitText
--> GetHttp --> Zip?

2. SplitText will get each of the line of text
3. The line feeds to GetHttp to property URL which it will download the file
and save it accordingly

Is this possible guys? It seems the properties of processor is immutable and
the only dynamic is when the expression language is allowed in the property.

Another question would be, the source text (dog.txt) it seems being executed
multiple times, how do I mark it that the NiFi got the inputs each line and
no repeat?

Thanks!!