You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2020/06/10 14:37:30 UTC
How to reference attribute by variable name
Hello. I currently have a python script in which I set values in a
dictionary from flowfile attributes, referencing them directly by name. I
then create a json string, which I can send to an AMQP broker as a message.
Something like this:
result['absolute.path'] = flowFile.getAttribute('absolute.path')
result['filename'] = flowFile.getAttribute('filename')
result['myAttr'] = flowFile.getAttribute('myAttr')
# Serialize the object to JSON...
json_str = json.dumps(result)
# Replace flowfile payload with this string representation of my json...
outputStream.write(unicode(json_str).encode('utf-8'))
I need to generalize this. I want to create attribute properties in my
ExecuteScript processor from which I call this python, each property being
a value I want to include in my json message.
Assuming I create an attribute property in the processor called draftPick,
and I set it to "Mahomes, KC". I read it into the script something like
this:
thisPick = *name of the property field*
.evaluateAttributeExpressions().getValue()
My first question: can I reference that incoming value in this expression
directly by variable reference without quoting as above, and is there a way
I can auto detect the property name to use as my key value in my dictionary?
result[*name of the property field*] = flowFile.getAttribute(*thisPick*)
My second question: is there a way to generalize this so that it
dynamically identifies and handles any variable number of properties and
values defined in my ExecuteScript processor? This would be the ideal
solution, because then I can have one python script that dynamically
handles any number of properties I set in the processor. I would do that
to grab whichever subset of attributes I happened to want to send to my
AMQP as a message at that time.
Jim
Re: How to reference attribute by variable name
Posted by James McMahon <js...@gmail.com>.
Very good. I will look into that more.
One way to solve this, I think, is for me to define anything I want to use
in an updateAttribute processor just prior to, and then I can reference in
Attributes List in AttributesToJSON (saving the JSON as flowfile content
for send to RabbitMQ).
Thanks for your thoughts and comments Mark. They are very helpful.
On Wed, Jun 10, 2020 at 11:06 AM Mark Payne <ma...@hotmail.com> wrote:
> Unfortunately, it doesn’t appear that the processor documents the
> variables that it provides. But it does provide access to the context, as
> the “context” variable:
>
> bindings.put("session", session);
> bindings.put("context", context);
> bindings.put("log", log);
> bindings.put("REL_SUCCESS", REL_SUCCESS);
> bindings.put("REL_FAILURE", REL_FAILURE);
>
> The JavaDocs for ProcessContext [1] can then be used to determine what’s
> available.
>
> Thanks
> -Mark
>
> [1]
> https://www.javadoc.io/doc/org.apache.nifi/nifi-api/1.11.0/org/apache/nifi/processor/ProcessContext.html
>
> On Jun 10, 2020, at 11:02 AM, James McMahon <js...@gmail.com> wrote:
>
> Mark, where can I find the methods that associate with ProcessContext? I
> can give it a try. I'm not smart enough to know how to tell whether a
> processor gives us access to ProcessContext.
>
> AttributesToJSON is an excellent "out of the box" solution when the values
> exist as attributes - I can then reference those in that comma-separated
> list of attributes to include in the JSON (alternatively it can include all
> attributes, which you already know). It wasn't clear to me, though, that a
> property I define in that same processor would be available to it to
> reference in property "Attributes List" in the same processor where it is
> defined as a property itself. Something like this:
> Attributes List .........................
> filename,absolute.path,newPropertyInProcessor
> .
> .
> .
> newPropertyInProcessor .......... aValue
>
> I'll test that. That would be simpler if I can reference new processor
> properties in the Attributes list. Thanks Mark.
>
> On Wed, Jun 10, 2020 at 10:47 AM Mark Payne <ma...@hotmail.com> wrote:
>
>> Jim,
>>
>> This isn’t a direct answer to your question. I don’t remember whether or
>> not ExecuteScript provides access directly to the ProcessContext or not -
>> if it does, then that provides methods for getting all properties and
>> determining whether or not the property is “dynamic” (aka user-defined).
>>
>> But given the description of this script, why not simply use
>> AttributesToJSON? It already provides a property for supplying a
>> comma-separated list of attributes and converts them into a JSON payload.
>>
>> Thanks
>> -Mark
>>
>>
>> On Jun 10, 2020, at 10:37 AM, James McMahon <js...@gmail.com> wrote:
>>
>> Hello. I currently have a python script in which I set values in a
>> dictionary from flowfile attributes, referencing them directly by name. I
>> then create a json string, which I can send to an AMQP broker as a message.
>> Something like this:
>> result['absolute.path'] = flowFile.getAttribute('absolute.path')
>> result['filename'] = flowFile.getAttribute('filename')
>> result['myAttr'] = flowFile.getAttribute('myAttr')
>> # Serialize the object to JSON...
>> json_str = json.dumps(result)
>> # Replace flowfile payload with this string representation of my json...
>> outputStream.write(unicode(json_str).encode('utf-8'))
>>
>> I need to generalize this. I want to create attribute properties in my
>> ExecuteScript processor from which I call this python, each property being
>> a value I want to include in my json message.
>>
>> Assuming I create an attribute property in the processor called
>> draftPick, and I set it to "Mahomes, KC". I read it into the script
>> something like this:
>> thisPick = *name of the property field*
>> .evaluateAttributeExpressions().getValue()
>>
>> My first question: can I reference that incoming value in this expression
>> directly by variable reference without quoting as above, and is there a way
>> I can auto detect the property name to use as my key value in my dictionary?
>> result[*name of the property field*] = flowFile.getAttribute(*thisPick*)
>>
>> My second question: is there a way to generalize this so that it
>> dynamically identifies and handles any variable number of properties and
>> values defined in my ExecuteScript processor? This would be the ideal
>> solution, because then I can have one python script that dynamically
>> handles any number of properties I set in the processor. I would do that
>> to grab whichever subset of attributes I happened to want to send to my
>> AMQP as a message at that time.
>>
>> Jim
>>
>>
>>
>
Re: How to reference attribute by variable name
Posted by Mark Payne <ma...@hotmail.com>.
Unfortunately, it doesn’t appear that the processor documents the variables that it provides. But it does provide access to the context, as the “context” variable:
bindings.put("session", session);
bindings.put("context", context);
bindings.put("log", log);
bindings.put("REL_SUCCESS", REL_SUCCESS);
bindings.put("REL_FAILURE", REL_FAILURE);
The JavaDocs for ProcessContext [1] can then be used to determine what’s available.
Thanks
-Mark
[1] https://www.javadoc.io/doc/org.apache.nifi/nifi-api/1.11.0/org/apache/nifi/processor/ProcessContext.html
On Jun 10, 2020, at 11:02 AM, James McMahon <js...@gmail.com>> wrote:
Mark, where can I find the methods that associate with ProcessContext? I can give it a try. I'm not smart enough to know how to tell whether a processor gives us access to ProcessContext.
AttributesToJSON is an excellent "out of the box" solution when the values exist as attributes - I can then reference those in that comma-separated list of attributes to include in the JSON (alternatively it can include all attributes, which you already know). It wasn't clear to me, though, that a property I define in that same processor would be available to it to reference in property "Attributes List" in the same processor where it is defined as a property itself. Something like this:
Attributes List ......................... filename,absolute.path,newPropertyInProcessor
.
.
.
newPropertyInProcessor .......... aValue
I'll test that. That would be simpler if I can reference new processor properties in the Attributes list. Thanks Mark.
On Wed, Jun 10, 2020 at 10:47 AM Mark Payne <ma...@hotmail.com>> wrote:
Jim,
This isn’t a direct answer to your question. I don’t remember whether or not ExecuteScript provides access directly to the ProcessContext or not - if it does, then that provides methods for getting all properties and determining whether or not the property is “dynamic” (aka user-defined).
But given the description of this script, why not simply use AttributesToJSON? It already provides a property for supplying a comma-separated list of attributes and converts them into a JSON payload.
Thanks
-Mark
On Jun 10, 2020, at 10:37 AM, James McMahon <js...@gmail.com>> wrote:
Hello. I currently have a python script in which I set values in a dictionary from flowfile attributes, referencing them directly by name. I then create a json string, which I can send to an AMQP broker as a message. Something like this:
result['absolute.path'] = flowFile.getAttribute('absolute.path')
result['filename'] = flowFile.getAttribute('filename')
result['myAttr'] = flowFile.getAttribute('myAttr')
# Serialize the object to JSON...
json_str = json.dumps(result)
# Replace flowfile payload with this string representation of my json...
outputStream.write(unicode(json_str).encode('utf-8'))
I need to generalize this. I want to create attribute properties in my ExecuteScript processor from which I call this python, each property being a value I want to include in my json message.
Assuming I create an attribute property in the processor called draftPick, and I set it to "Mahomes, KC". I read it into the script something like this:
thisPick = name of the property field .evaluateAttributeExpressions().getValue()
My first question: can I reference that incoming value in this expression directly by variable reference without quoting as above, and is there a way I can auto detect the property name to use as my key value in my dictionary?
result[name of the property field] = flowFile.getAttribute(thisPick)
My second question: is there a way to generalize this so that it dynamically identifies and handles any variable number of properties and values defined in my ExecuteScript processor? This would be the ideal solution, because then I can have one python script that dynamically handles any number of properties I set in the processor. I would do that to grab whichever subset of attributes I happened to want to send to my AMQP as a message at that time.
Jim
Re: How to reference attribute by variable name
Posted by James McMahon <js...@gmail.com>.
Mark, where can I find the methods that associate with ProcessContext? I
can give it a try. I'm not smart enough to know how to tell whether a
processor gives us access to ProcessContext.
AttributesToJSON is an excellent "out of the box" solution when the values
exist as attributes - I can then reference those in that comma-separated
list of attributes to include in the JSON (alternatively it can include all
attributes, which you already know). It wasn't clear to me, though, that a
property I define in that same processor would be available to it to
reference in property "Attributes List" in the same processor where it is
defined as a property itself. Something like this:
Attributes List .........................
filename,absolute.path,newPropertyInProcessor
.
.
.
newPropertyInProcessor .......... aValue
I'll test that. That would be simpler if I can reference new processor
properties in the Attributes list. Thanks Mark.
On Wed, Jun 10, 2020 at 10:47 AM Mark Payne <ma...@hotmail.com> wrote:
> Jim,
>
> This isn’t a direct answer to your question. I don’t remember whether or
> not ExecuteScript provides access directly to the ProcessContext or not -
> if it does, then that provides methods for getting all properties and
> determining whether or not the property is “dynamic” (aka user-defined).
>
> But given the description of this script, why not simply use
> AttributesToJSON? It already provides a property for supplying a
> comma-separated list of attributes and converts them into a JSON payload.
>
> Thanks
> -Mark
>
>
> On Jun 10, 2020, at 10:37 AM, James McMahon <js...@gmail.com> wrote:
>
> Hello. I currently have a python script in which I set values in a
> dictionary from flowfile attributes, referencing them directly by name. I
> then create a json string, which I can send to an AMQP broker as a message.
> Something like this:
> result['absolute.path'] = flowFile.getAttribute('absolute.path')
> result['filename'] = flowFile.getAttribute('filename')
> result['myAttr'] = flowFile.getAttribute('myAttr')
> # Serialize the object to JSON...
> json_str = json.dumps(result)
> # Replace flowfile payload with this string representation of my json...
> outputStream.write(unicode(json_str).encode('utf-8'))
>
> I need to generalize this. I want to create attribute properties in my
> ExecuteScript processor from which I call this python, each property being
> a value I want to include in my json message.
>
> Assuming I create an attribute property in the processor called draftPick,
> and I set it to "Mahomes, KC". I read it into the script something like
> this:
> thisPick = *name of the property field*
> .evaluateAttributeExpressions().getValue()
>
> My first question: can I reference that incoming value in this expression
> directly by variable reference without quoting as above, and is there a way
> I can auto detect the property name to use as my key value in my dictionary?
> result[*name of the property field*] = flowFile.getAttribute(*thisPick*)
>
> My second question: is there a way to generalize this so that it
> dynamically identifies and handles any variable number of properties and
> values defined in my ExecuteScript processor? This would be the ideal
> solution, because then I can have one python script that dynamically
> handles any number of properties I set in the processor. I would do that
> to grab whichever subset of attributes I happened to want to send to my
> AMQP as a message at that time.
>
> Jim
>
>
>
Re: How to reference attribute by variable name
Posted by Mark Payne <ma...@hotmail.com>.
Jim,
This isn’t a direct answer to your question. I don’t remember whether or not ExecuteScript provides access directly to the ProcessContext or not - if it does, then that provides methods for getting all properties and determining whether or not the property is “dynamic” (aka user-defined).
But given the description of this script, why not simply use AttributesToJSON? It already provides a property for supplying a comma-separated list of attributes and converts them into a JSON payload.
Thanks
-Mark
On Jun 10, 2020, at 10:37 AM, James McMahon <js...@gmail.com>> wrote:
Hello. I currently have a python script in which I set values in a dictionary from flowfile attributes, referencing them directly by name. I then create a json string, which I can send to an AMQP broker as a message. Something like this:
result['absolute.path'] = flowFile.getAttribute('absolute.path')
result['filename'] = flowFile.getAttribute('filename')
result['myAttr'] = flowFile.getAttribute('myAttr')
# Serialize the object to JSON...
json_str = json.dumps(result)
# Replace flowfile payload with this string representation of my json...
outputStream.write(unicode(json_str).encode('utf-8'))
I need to generalize this. I want to create attribute properties in my ExecuteScript processor from which I call this python, each property being a value I want to include in my json message.
Assuming I create an attribute property in the processor called draftPick, and I set it to "Mahomes, KC". I read it into the script something like this:
thisPick = name of the property field .evaluateAttributeExpressions().getValue()
My first question: can I reference that incoming value in this expression directly by variable reference without quoting as above, and is there a way I can auto detect the property name to use as my key value in my dictionary?
result[name of the property field] = flowFile.getAttribute(thisPick)
My second question: is there a way to generalize this so that it dynamically identifies and handles any variable number of properties and values defined in my ExecuteScript processor? This would be the ideal solution, because then I can have one python script that dynamically handles any number of properties I set in the processor. I would do that to grab whichever subset of attributes I happened to want to send to my AMQP as a message at that time.
Jim