You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2017/07/12 01:12:53 UTC
KeyError from Jython in ExecuteScript
Good evening. I am in a Jython 2.7.0 environment, using NiFi 1.3.0.
In my script I check for validity in dictionary key values, like so:
if (
result['application'] in valid_application
and result['environment'] in
valid_environment
and result['fromComponent'] in valid_from
and result['toComponent'] in valid_to
and result['messageComponent'] in
valid_messageComponent
and result['messageTypeState'] in
valid_messageTypeState
and result['messageType'] in
valid_messageType
) :
flowFile = session.putAttribute(flowFile,
'isValidMessage', 'True')
else :
flowFile = session.putAttribute(flowFile,
'isValidMessage', 'False')
I curl an Http POST that populates all the key values. When they are all
there, the script works wonderfully. But when any one has no value in the
incoming json submitted via POST, it throws an error. For example, like so
, "messageComponent":""
The error thrown and logged by NiFi to nifi-app.log is this:
2017-07-12 01:03:55,578 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.script.ExecuteScript
ExecuteScript[id=61381fb9-1012-115d-2a56-9f9e7fe9f382] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException:
javax.script.ScriptException: KeyError: 'environment' in <script> at line
number 254: {}
org.apache.nifi.processor.exception.ProcessException:
javax.script.ScriptException: KeyError: 'environment' in <script> at line
number 254
at
org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:230)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.script.ScriptException: KeyError: 'environment' in
<script> at line number 254
at
org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:202)
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
at
javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at
org.apache.nifi.script.impl.JythonScriptEngineConfigurator.eval(JythonScriptEngineConfigurator.java:59)
at
org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:220)
... 11 common frames omitted
Caused by: org.python.core.PyException: null
at org.python.core.Py.KeyError(Py.java:249)
at org.python.core.PyObject.__getitem__(PyObject.java:738)
at org.python.pycode._pyx15333737.f$0(<script>:245)
at org.python.pycode._pyx15333737.call_function(<script>)
at org.python.core.PyTableCode.call(PyTableCode.java:167)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1386)
at org.python.core.__builtin__.eval(__builtin__.java:497)
at org.python.core.__builtin__.eval(__builtin__.java:501)
at
org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
... 15 common frames omitted
What can I do to avoid this error, since I am unable to determine why the
empty incoming value is causing it to occur?
Thanks in advance for your help.
Cheers,
Jim
Re: KeyError from Jython in ExecuteScript
Posted by James McMahon <js...@gmail.com>.
Many thanks to you guys for your suggestions. With your help I was able to
overcome this challenge. Here is the section of code that I modified:
flowFiles = session.get(20)
for flowFile in flowFiles :
if (flowFile != None) :
flowFile = session.write(flowFile,PyStreamCallback())
# only get back result with length 1 if invalid JSON...
if (len(result) == 1) :
session.transfer(flowFile, REL_FAILURE)
else :
if (
result.get('application','x') in
valid_application
and result.get('environment','x') in
valid_environment
and result.get('fromComponent','x') in
valid_from
and result.get('toComponent','x') in
valid_to
and result.get('messageComponent','x') in
valid_messageComponent
and result.get('messageTypeState','x') in
valid_messageTypeState
and result.get('messageType','x') in
valid_messageType
) :
flowFile = session.putAttribute(flowFile,
'isValidMessage', 'True')
else :
flowFile = session.putAttribute(flowFile,
'isValidMessage', 'False')
Attribute isValidMessage is later used by RouteAttribute to dictate the
course to an HandleHttpResponse - to a 202 if the incoming message is
valid, to a 400 if the incoming json message lacks required key-value pairs.
Thanks again.
Jim
On Wed, Jul 12, 2017 at 1:04 PM, James McMahon <js...@gmail.com> wrote:
> Thanks very much Scott. I intend to try the logic improvements you, Andy,
> and Matt have discussed and will circle back with the results tomorrow. -Jim
>
> On Wed, Jul 12, 2017 at 12:46 PM, Scott Wagner <sw...@beenverified.com>
> wrote:
>
>> Python/Jython dictionaries also support a get() method that allows you to
>> provide a value to use if the key is not in the dictionary.
>>
>> The question is what do you want to have happen if your result dictionary
>> doesn't contain the value? Do you want it to continue on, or do you want
>> it to fail?
>>
>> if result.get('application', '') in valid_application and
>> result.get('environment', '') in valid_environment ...
>>
>> If you want it to continue on if they are missing, then add an empty
>> string to each of the sets you are comparing. Also, if an empty string is
>> a possible input and you want to have the missing value be different, you
>> can specify a different default as the second argument to the get() method.
>>
>> The KeyError is thrown whenever you try to reference a key that isn't in
>> the dictionary. This page[1] might provide you some more helpful details.
>>
>> - Scott
>>
>> [1] https://wiki.python.org/moin/KeyError
>>
>> James McMahon <js...@gmail.com>
>> Wednesday, July 12, 2017 4:43 AM
>> Thank you very much, Matt and Andy. I will explore these suggested
>> improvements to my logic this evening and post the results. Cheers, Jim
>>
>>
>> Matt Burgess <ma...@apache.org>
>> Tuesday, July 11, 2017 8:25 PM
>> Jim,
>>
>> You can check first if the key is in the results dictionary and then
>> if the value is in the valid values dictionary:
>>
>> ...
>> and 'environment' in result and result['environment'] in valid_environment
>> ...
>>
>> if a missing value is "valid", then you can use "or" instead of "and"
>> and put the clause in parens:
>>
>> ...
>> and ('environment' in result or result['environment'] in
>> valid_environment)
>> ...
>>
>>
>> If I misunderstood what you were asking please let me know and I'll try
>> again :)
>>
>> Regards,
>> Matt
>>
>> James McMahon <js...@gmail.com>
>> Tuesday, July 11, 2017 8:12 PM
>> Good evening. I am in a Jython 2.7.0 environment, using NiFi 1.3.0.
>>
>> In my script I check for validity in dictionary key values, like so:
>>
>> if (
>> result['application'] in valid_application
>> and result['environment'] in
>> valid_environment
>> and result['fromComponent'] in valid_from
>> and result['toComponent'] in valid_to
>> and result['messageComponent'] in
>> valid_messageComponent
>> and result['messageTypeState'] in
>> valid_messageTypeState
>> and result['messageType'] in
>> valid_messageType
>> ) :
>> flowFile = session.putAttribute(flowFile,
>> 'isValidMessage', 'True')
>> else :
>> flowFile = session.putAttribute(flowFile,
>> 'isValidMessage', 'False')
>>
>> I curl an Http POST that populates all the key values. When they are all
>> there, the script works wonderfully. But when any one has no value in the
>> incoming json submitted via POST, it throws an error. For example, like so
>> , "messageComponent":""
>>
>> The error thrown and logged by NiFi to nifi-app.log is this:
>>
>> 2017-07-12 01:03:55,578 ERROR [Timer-Driven Process Thread-3]
>> o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=61381fb9-1012-115d-2a56-9f9e7fe9f382]
>> Failed to process session due to org.apache.nifi.processor.exception.ProcessException:
>> javax.script.ScriptException: KeyError: 'environment' in <script> at line
>> number 254: {}
>> org.apache.nifi.processor.exception.ProcessException:
>> javax.script.ScriptException: KeyError: 'environment' in <script> at line
>> number 254
>> at org.apache.nifi.processors.script.ExecuteScript.onTrigger(
>> ExecuteScript.java:230)
>> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>> tandardProcessorNode.java:1120)
>> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:147)
>> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:47)
>> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>> gent$1.run(TimerDrivenSchedulingAgent.java:132)
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:511)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:
>> 308)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: javax.script.ScriptException: KeyError: 'environment' in
>> <script> at line number 254
>> at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEng
>> ine.java:202)
>> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
>> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
>> at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.
>> java:264)
>> at org.apache.nifi.script.impl.JythonScriptEngineConfigurator.
>> eval(JythonScriptEngineConfigurator.java:59)
>> at org.apache.nifi.processors.script.ExecuteScript.onTrigger(
>> ExecuteScript.java:220)
>> ... 11 common frames omitted
>> Caused by: org.python.core.PyException: null
>> at org.python.core.Py.KeyError(Py.java:249)
>> at org.python.core.PyObject.__getitem__(PyObject.java:738)
>> at org.python.pycode._pyx15333737.f$0(<script>:245)
>> at org.python.pycode._pyx15333737.call_function(<script>)
>> at org.python.core.PyTableCode.call(PyTableCode.java:167)
>> at org.python.core.PyCode.call(PyCode.java:18)
>> at org.python.core.Py.runCode(Py.java:1386)
>> at org.python.core.__builtin__.eval(__builtin__.java:497)
>> at org.python.core.__builtin__.eval(__builtin__.java:501)
>> at org.python.util.PythonInterpreter.eval(PythonInterpreter.
>> java:259)
>> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
>> ... 15 common frames omitted
>>
>> What can I do to avoid this error, since I am unable to determine why the
>> empty incoming value is causing it to occur?
>>
>> Thanks in advance for your help.
>>
>> Cheers,
>>
>> Jim
>>
>>
>>
>
Re: KeyError from Jython in ExecuteScript
Posted by James McMahon <js...@gmail.com>.
Thanks very much Scott. I intend to try the logic improvements you, Andy,
and Matt have discussed and will circle back with the results tomorrow. -Jim
On Wed, Jul 12, 2017 at 12:46 PM, Scott Wagner <sw...@beenverified.com>
wrote:
> Python/Jython dictionaries also support a get() method that allows you to
> provide a value to use if the key is not in the dictionary.
>
> The question is what do you want to have happen if your result dictionary
> doesn't contain the value? Do you want it to continue on, or do you want
> it to fail?
>
> if result.get('application', '') in valid_application and
> result.get('environment', '') in valid_environment ...
>
> If you want it to continue on if they are missing, then add an empty
> string to each of the sets you are comparing. Also, if an empty string is
> a possible input and you want to have the missing value be different, you
> can specify a different default as the second argument to the get() method.
>
> The KeyError is thrown whenever you try to reference a key that isn't in
> the dictionary. This page[1] might provide you some more helpful details.
>
> - Scott
>
> [1] https://wiki.python.org/moin/KeyError
>
> James McMahon <js...@gmail.com>
> Wednesday, July 12, 2017 4:43 AM
> Thank you very much, Matt and Andy. I will explore these suggested
> improvements to my logic this evening and post the results. Cheers, Jim
>
>
> Matt Burgess <ma...@apache.org>
> Tuesday, July 11, 2017 8:25 PM
> Jim,
>
> You can check first if the key is in the results dictionary and then
> if the value is in the valid values dictionary:
>
> ...
> and 'environment' in result and result['environment'] in valid_environment
> ...
>
> if a missing value is "valid", then you can use "or" instead of "and"
> and put the clause in parens:
>
> ...
> and ('environment' in result or result['environment'] in valid_environment)
> ...
>
>
> If I misunderstood what you were asking please let me know and I'll try
> again :)
>
> Regards,
> Matt
>
> James McMahon <js...@gmail.com>
> Tuesday, July 11, 2017 8:12 PM
> Good evening. I am in a Jython 2.7.0 environment, using NiFi 1.3.0.
>
> In my script I check for validity in dictionary key values, like so:
>
> if (
> result['application'] in valid_application
> and result['environment'] in
> valid_environment
> and result['fromComponent'] in valid_from
> and result['toComponent'] in valid_to
> and result['messageComponent'] in
> valid_messageComponent
> and result['messageTypeState'] in
> valid_messageTypeState
> and result['messageType'] in
> valid_messageType
> ) :
> flowFile = session.putAttribute(flowFile,
> 'isValidMessage', 'True')
> else :
> flowFile = session.putAttribute(flowFile,
> 'isValidMessage', 'False')
>
> I curl an Http POST that populates all the key values. When they are all
> there, the script works wonderfully. But when any one has no value in the
> incoming json submitted via POST, it throws an error. For example, like so
> , "messageComponent":""
>
> The error thrown and logged by NiFi to nifi-app.log is this:
>
> 2017-07-12 01:03:55,578 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=61381fb9-1012-115d-2a56-9f9e7fe9f382]
> Failed to process session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: KeyError: 'environment' in <script> at line
> number 254: {}
> org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: KeyError: 'environment' in <script> at line
> number 254
> at org.apache.nifi.processors.script.ExecuteScript.
> onTrigger(ExecuteScript.java:230)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1120)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:147)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:47)
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.script.ScriptException: KeyError: 'environment' in
> <script> at line number 254
> at org.python.jsr223.PyScriptEngine.scriptException(
> PyScriptEngine.java:202)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
> at javax.script.AbstractScriptEngine.eval(
> AbstractScriptEngine.java:264)
> at org.apache.nifi.script.impl.JythonScriptEngineConfigurator
> .eval(JythonScriptEngineConfigurator.java:59)
> at org.apache.nifi.processors.script.ExecuteScript.
> onTrigger(ExecuteScript.java:220)
> ... 11 common frames omitted
> Caused by: org.python.core.PyException: null
> at org.python.core.Py.KeyError(Py.java:249)
> at org.python.core.PyObject.__getitem__(PyObject.java:738)
> at org.python.pycode._pyx15333737.f$0(<script>:245)
> at org.python.pycode._pyx15333737.call_function(<script>)
> at org.python.core.PyTableCode.call(PyTableCode.java:167)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1386)
> at org.python.core.__builtin__.eval(__builtin__.java:497)
> at org.python.core.__builtin__.eval(__builtin__.java:501)
> at org.python.util.PythonInterpreter.eval(
> PythonInterpreter.java:259)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
> ... 15 common frames omitted
>
> What can I do to avoid this error, since I am unable to determine why the
> empty incoming value is causing it to occur?
>
> Thanks in advance for your help.
>
> Cheers,
>
> Jim
>
>
>
Re: KeyError from Jython in ExecuteScript
Posted by Scott Wagner <sw...@beenverified.com>.
Python/Jython dictionaries also support a get() method that allows you
to provide a value to use if the key is not in the dictionary.
The question is what do you want to have happen if your result
dictionary doesn't contain the value? Do you want it to continue on, or
do you want it to fail?
if result.get('application', '') in valid_application and
result.get('environment', '') in valid_environment ...
If you want it to continue on if they are missing, then add an empty
string to each of the sets you are comparing. Also, if an empty string
is a possible input and you want to have the missing value be different,
you can specify a different default as the second argument to the get()
method.
The KeyError is thrown whenever you try to reference a key that isn't in
the dictionary. This page[1] might provide you some more helpful details.
- Scott
[1] https://wiki.python.org/moin/KeyError
> James McMahon <ma...@gmail.com>
> Wednesday, July 12, 2017 4:43 AM
> Thank you very much, Matt and Andy. I will explore these suggested
> improvements to my logic this evening and post the results. Cheers, Jim
>
>
> Matt Burgess <ma...@apache.org>
> Tuesday, July 11, 2017 8:25 PM
> Jim,
>
> You can check first if the key is in the results dictionary and then
> if the value is in the valid values dictionary:
>
> ...
> and 'environment' in result and result['environment'] in valid_environment
> ...
>
> if a missing value is "valid", then you can use "or" instead of "and"
> and put the clause in parens:
>
> ...
> and ('environment' in result or result['environment'] in
> valid_environment)
> ...
>
>
> If I misunderstood what you were asking please let me know and I'll
> try again :)
>
> Regards,
> Matt
>
> James McMahon <ma...@gmail.com>
> Tuesday, July 11, 2017 8:12 PM
> Good evening. I am in a Jython 2.7.0 environment, using NiFi 1.3.0.
>
> In my script I check for validity in dictionary key values, like so:
>
> if (
> result['application'] in valid_application
> and result['environment'] in
> valid_environment
> and result['fromComponent'] in valid_from
> and result['toComponent'] in valid_to
> and result['messageComponent'] in
> valid_messageComponent
> and result['messageTypeState'] in
> valid_messageTypeState
> and result['messageType'] in
> valid_messageType
> ) :
> flowFile =
> session.putAttribute(flowFile, 'isValidMessage', 'True')
> else :
> flowFile =
> session.putAttribute(flowFile, 'isValidMessage', 'False')
>
> I curl an Http POST that populates all the key values. When they are
> all there, the script works wonderfully. But when any one has no value
> in the incoming json submitted via POST, it throws an error. For
> example, like so , "messageComponent":""
>
> The error thrown and logged by NiFi to nifi-app.log is this:
>
> 2017-07-12 01:03:55,578 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.script.ExecuteScript
> ExecuteScript[id=61381fb9-1012-115d-2a56-9f9e7fe9f382] Failed to
> process session due to
> org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: KeyError: 'environment' in <script> at
> line number 254: {}
> org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: KeyError: 'environment' in <script> at
> line number 254
> at
> org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:230)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.script.ScriptException: KeyError: 'environment' in
> <script> at line number 254
> at
> org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:202)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
> at
> javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> at
> org.apache.nifi.script.impl.JythonScriptEngineConfigurator.eval(JythonScriptEngineConfigurator.java:59)
> at
> org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:220)
> ... 11 common frames omitted
> Caused by: org.python.core.PyException: null
> at org.python.core.Py.KeyError(Py.java:249)
> at org.python.core.PyObject.__getitem__(PyObject.java:738)
> at org.python.pycode._pyx15333737.f$0(<script>:245)
> at org.python.pycode._pyx15333737.call_function(<script>)
> at org.python.core.PyTableCode.call(PyTableCode.java:167)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1386)
> at org.python.core.__builtin__.eval(__builtin__.java:497)
> at org.python.core.__builtin__.eval(__builtin__.java:501)
> at
> org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
> ... 15 common frames omitted
>
> What can I do to avoid this error, since I am unable to determine why
> the empty incoming value is causing it to occur?
>
> Thanks in advance for your help.
>
> Cheers,
>
> Jim
Re: KeyError from Jython in ExecuteScript
Posted by James McMahon <js...@gmail.com>.
Thank you very much, Matt and Andy. I will explore these suggested
improvements to my logic this evening and post the results. Cheers, Jim
On Tue, Jul 11, 2017 at 9:25 PM, Matt Burgess <ma...@apache.org> wrote:
> Jim,
>
> You can check first if the key is in the results dictionary and then
> if the value is in the valid values dictionary:
>
> ...
> and 'environment' in result and result['environment'] in valid_environment
> ...
>
> if a missing value is "valid", then you can use "or" instead of "and"
> and put the clause in parens:
>
> ...
> and ('environment' in result or result['environment'] in valid_environment)
> ...
>
>
> If I misunderstood what you were asking please let me know and I'll try
> again :)
>
> Regards,
> Matt
>
>
> On Tue, Jul 11, 2017 at 9:12 PM, James McMahon <js...@gmail.com>
> wrote:
> > Good evening. I am in a Jython 2.7.0 environment, using NiFi 1.3.0.
> >
> > In my script I check for validity in dictionary key values, like so:
> >
> > if (
> > result['application'] in
> valid_application
> > and result['environment'] in
> > valid_environment
> > and result['fromComponent'] in valid_from
> > and result['toComponent'] in valid_to
> > and result['messageComponent'] in
> > valid_messageComponent
> > and result['messageTypeState'] in
> > valid_messageTypeState
> > and result['messageType'] in
> > valid_messageType
> > ) :
> > flowFile = session.putAttribute(flowFile,
> > 'isValidMessage', 'True')
> > else :
> > flowFile = session.putAttribute(flowFile,
> > 'isValidMessage', 'False')
> >
> > I curl an Http POST that populates all the key values. When they are all
> > there, the script works wonderfully. But when any one has no value in the
> > incoming json submitted via POST, it throws an error. For example, like
> so
> > , "messageComponent":""
> >
> > The error thrown and logged by NiFi to nifi-app.log is this:
> >
> > 2017-07-12 01:03:55,578 ERROR [Timer-Driven Process Thread-3]
> > o.a.nifi.processors.script.ExecuteScript
> > ExecuteScript[id=61381fb9-1012-115d-2a56-9f9e7fe9f382] Failed to process
> > session due to org.apache.nifi.processor.exception.ProcessException:
> > javax.script.ScriptException: KeyError: 'environment' in <script> at line
> > number 254: {}
> > org.apache.nifi.processor.exception.ProcessException:
> > javax.script.ScriptException: KeyError: 'environment' in <script> at line
> > number 254
> > at
> > org.apache.nifi.processors.script.ExecuteScript.
> onTrigger(ExecuteScript.java:230)
> > at
> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1120)
> > at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:147)
> > at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:47)
> > at
> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> TimerDrivenSchedulingAgent.java:132)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: javax.script.ScriptException: KeyError: 'environment' in
> <script>
> > at line number 254
> > at
> > org.python.jsr223.PyScriptEngine.scriptException(
> PyScriptEngine.java:202)
> > at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
> > at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
> > at
> > javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> > at
> > org.apache.nifi.script.impl.JythonScriptEngineConfigurator.eval(
> JythonScriptEngineConfigurator.java:59)
> > at
> > org.apache.nifi.processors.script.ExecuteScript.
> onTrigger(ExecuteScript.java:220)
> > ... 11 common frames omitted
> > Caused by: org.python.core.PyException: null
> > at org.python.core.Py.KeyError(Py.java:249)
> > at org.python.core.PyObject.__getitem__(PyObject.java:738)
> > at org.python.pycode._pyx15333737.f$0(<script>:245)
> > at org.python.pycode._pyx15333737.call_function(<script>)
> > at org.python.core.PyTableCode.call(PyTableCode.java:167)
> > at org.python.core.PyCode.call(PyCode.java:18)
> > at org.python.core.Py.runCode(Py.java:1386)
> > at org.python.core.__builtin__.eval(__builtin__.java:497)
> > at org.python.core.__builtin__.eval(__builtin__.java:501)
> > at
> > org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
> > at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
> > ... 15 common frames omitted
> >
> > What can I do to avoid this error, since I am unable to determine why the
> > empty incoming value is causing it to occur?
> >
> > Thanks in advance for your help.
> >
> > Cheers,
> >
> > Jim
>
Re: KeyError from Jython in ExecuteScript
Posted by Matt Burgess <ma...@apache.org>.
Jim,
You can check first if the key is in the results dictionary and then
if the value is in the valid values dictionary:
...
and 'environment' in result and result['environment'] in valid_environment
...
if a missing value is "valid", then you can use "or" instead of "and"
and put the clause in parens:
...
and ('environment' in result or result['environment'] in valid_environment)
...
If I misunderstood what you were asking please let me know and I'll try again :)
Regards,
Matt
On Tue, Jul 11, 2017 at 9:12 PM, James McMahon <js...@gmail.com> wrote:
> Good evening. I am in a Jython 2.7.0 environment, using NiFi 1.3.0.
>
> In my script I check for validity in dictionary key values, like so:
>
> if (
> result['application'] in valid_application
> and result['environment'] in
> valid_environment
> and result['fromComponent'] in valid_from
> and result['toComponent'] in valid_to
> and result['messageComponent'] in
> valid_messageComponent
> and result['messageTypeState'] in
> valid_messageTypeState
> and result['messageType'] in
> valid_messageType
> ) :
> flowFile = session.putAttribute(flowFile,
> 'isValidMessage', 'True')
> else :
> flowFile = session.putAttribute(flowFile,
> 'isValidMessage', 'False')
>
> I curl an Http POST that populates all the key values. When they are all
> there, the script works wonderfully. But when any one has no value in the
> incoming json submitted via POST, it throws an error. For example, like so
> , "messageComponent":""
>
> The error thrown and logged by NiFi to nifi-app.log is this:
>
> 2017-07-12 01:03:55,578 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.script.ExecuteScript
> ExecuteScript[id=61381fb9-1012-115d-2a56-9f9e7fe9f382] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: KeyError: 'environment' in <script> at line
> number 254: {}
> org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: KeyError: 'environment' in <script> at line
> number 254
> at
> org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:230)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.script.ScriptException: KeyError: 'environment' in <script>
> at line number 254
> at
> org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:202)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
> at
> javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> at
> org.apache.nifi.script.impl.JythonScriptEngineConfigurator.eval(JythonScriptEngineConfigurator.java:59)
> at
> org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:220)
> ... 11 common frames omitted
> Caused by: org.python.core.PyException: null
> at org.python.core.Py.KeyError(Py.java:249)
> at org.python.core.PyObject.__getitem__(PyObject.java:738)
> at org.python.pycode._pyx15333737.f$0(<script>:245)
> at org.python.pycode._pyx15333737.call_function(<script>)
> at org.python.core.PyTableCode.call(PyTableCode.java:167)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1386)
> at org.python.core.__builtin__.eval(__builtin__.java:497)
> at org.python.core.__builtin__.eval(__builtin__.java:501)
> at
> org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
> ... 15 common frames omitted
>
> What can I do to avoid this error, since I am unable to determine why the
> empty incoming value is causing it to occur?
>
> Thanks in advance for your help.
>
> Cheers,
>
> Jim
Re: KeyError from Jython in ExecuteScript
Posted by James McMahon <js...@gmail.com>.
I should add that valid_... is defined in my script in this manner, using
environment as this example:
valid_environment = ['development', 'integration', 'test', 'production']
On Tue, Jul 11, 2017 at 9:12 PM, James McMahon <js...@gmail.com> wrote:
> Good evening. I am in a Jython 2.7.0 environment, using NiFi 1.3.0.
>
> In my script I check for validity in dictionary key values, like so:
>
> if (
> result['application'] in valid_application
> and result['environment'] in
> valid_environment
> and result['fromComponent'] in valid_from
> and result['toComponent'] in valid_to
> and result['messageComponent'] in
> valid_messageComponent
> and result['messageTypeState'] in
> valid_messageTypeState
> and result['messageType'] in
> valid_messageType
> ) :
> flowFile = session.putAttribute(flowFile,
> 'isValidMessage', 'True')
> else :
> flowFile = session.putAttribute(flowFile,
> 'isValidMessage', 'False')
>
> I curl an Http POST that populates all the key values. When they are all
> there, the script works wonderfully. But when any one has no value in the
> incoming json submitted via POST, it throws an error. For example, like so
> , "messageComponent":""
>
> The error thrown and logged by NiFi to nifi-app.log is this:
>
> 2017-07-12 01:03:55,578 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=61381fb9-1012-115d-2a56-9f9e7fe9f382]
> Failed to process session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: KeyError: 'environment' in <script> at line
> number 254: {}
> org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: KeyError: 'environment' in <script> at line
> number 254
> at org.apache.nifi.processors.script.ExecuteScript.
> onTrigger(ExecuteScript.java:230)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1120)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:147)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:47)
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.script.ScriptException: KeyError: 'environment' in
> <script> at line number 254
> at org.python.jsr223.PyScriptEngine.scriptException(
> PyScriptEngine.java:202)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
> at javax.script.AbstractScriptEngine.eval(
> AbstractScriptEngine.java:264)
> at org.apache.nifi.script.impl.JythonScriptEngineConfigurator
> .eval(JythonScriptEngineConfigurator.java:59)
> at org.apache.nifi.processors.script.ExecuteScript.
> onTrigger(ExecuteScript.java:220)
> ... 11 common frames omitted
> Caused by: org.python.core.PyException: null
> at org.python.core.Py.KeyError(Py.java:249)
> at org.python.core.PyObject.__getitem__(PyObject.java:738)
> at org.python.pycode._pyx15333737.f$0(<script>:245)
> at org.python.pycode._pyx15333737.call_function(<script>)
> at org.python.core.PyTableCode.call(PyTableCode.java:167)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1386)
> at org.python.core.__builtin__.eval(__builtin__.java:497)
> at org.python.core.__builtin__.eval(__builtin__.java:501)
> at org.python.util.PythonInterpreter.eval(
> PythonInterpreter.java:259)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
> ... 15 common frames omitted
>
> What can I do to avoid this error, since I am unable to determine why the
> empty incoming value is causing it to occur?
>
> Thanks in advance for your help.
>
> Cheers,
>
> Jim
>
Re: KeyError from Jython in ExecuteScript
Posted by Andy LoPresto <al...@apache.org>.
James,
I believe this is because if the key doesn’t exist, Python returns null, and then tries to compare null to whatever datatype valid_XXX is. You can see in the stacktrace that there is a "org.python.core.PyException: at org.python.core.Py.KeyError(Py.java:249)”. I’m not a Python expert, but that’s my guess. To avoid this, you probably need to check the existence of the value before performing the comparison (i.e. in Java “if (variable != null && someList.contains(variable))”).
Perhaps another user is more knowledgable about proper Python syntax. It appears you can also use “get()” [1] to access a key and return a default value rather than an exception if the key is not present.
[1] https://wiki.python.org/moin/KeyError <https://wiki.python.org/moin/KeyError>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jul 11, 2017, at 6:12 PM, James McMahon <js...@gmail.com> wrote:
>
> Good evening. I am in a Jython 2.7.0 environment, using NiFi 1.3.0.
>
> In my script I check for validity in dictionary key values, like so:
>
> if (
> result['application'] in valid_application
> and result['environment'] in valid_environment
> and result['fromComponent'] in valid_from
> and result['toComponent'] in valid_to
> and result['messageComponent'] in valid_messageComponent
> and result['messageTypeState'] in valid_messageTypeState
> and result['messageType'] in valid_messageType
> ) :
> flowFile = session.putAttribute(flowFile, 'isValidMessage', 'True')
> else :
> flowFile = session.putAttribute(flowFile, 'isValidMessage', 'False')
>
> I curl an Http POST that populates all the key values. When they are all there, the script works wonderfully. But when any one has no value in the incoming json submitted via POST, it throws an error. For example, like so , "messageComponent":""
>
> The error thrown and logged by NiFi to nifi-app.log is this:
>
> 2017-07-12 01:03:55,578 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=61381fb9-1012-115d-2a56-9f9e7fe9f382] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: KeyError: 'environment' in <script> at line number 254: {}
> org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: KeyError: 'environment' in <script> at line number 254
> at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:230)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.script.ScriptException: KeyError: 'environment' in <script> at line number 254
> at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:202)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
> at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
> at org.apache.nifi.script.impl.JythonScriptEngineConfigurator.eval(JythonScriptEngineConfigurator.java:59)
> at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:220)
> ... 11 common frames omitted
> Caused by: org.python.core.PyException: null
> at org.python.core.Py.KeyError(Py.java:249)
> at org.python.core.PyObject.__getitem__(PyObject.java:738)
> at org.python.pycode._pyx15333737.f$0(<script>:245)
> at org.python.pycode._pyx15333737.call_function(<script>)
> at org.python.core.PyTableCode.call(PyTableCode.java:167)
> at org.python.core.PyCode.call(PyCode.java:18)
> at org.python.core.Py.runCode(Py.java:1386)
> at org.python.core.__builtin__.eval(__builtin__.java:497)
> at org.python.core.__builtin__.eval(__builtin__.java:501)
> at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
> at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40)
> ... 15 common frames omitted
>
> What can I do to avoid this error, since I am unable to determine why the empty incoming value is causing it to occur?
>
> Thanks in advance for your help.
>
> Cheers,
>
> Jim