You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mandeep Gill <ma...@nstack.com> on 2018/11/13 15:16:20 UTC
JsonPath expression language exception with QueryRecord processor
Hi there,
I'm hitting an issue using the `jsonPath` expression language function to
extract a query to use with the QueryRecord processor. The processor works
fine if the expression language subject is contained within the process
group variable registry, but fails upon starting with an
AttributeExpressionLanguageException if the subject was expected to to
exist within a flowfile attribute [1]
I've attached a template generated on NiFi 1.8.0 demonstrating the problem
- it only appears to be an issue with dynamic outputs from the QueryRecord
processors, as the same expression language statement works fine when used
as part of UpdateAttribute processor with the subject in a flowfile
attribute as per the template. I've dug into the codebase and can trace the
error to the evaluate function within the JsonPathEvaluator class, which
throws the exception if the variable can not be referenced. I have a local
fix at https://github.com/apache/nifi/compare/master...nstack:fix/jsonpath that
returns StringQueryResult("") if the subject is empty instead of throwing
the exception and this appears to work however I wonder that as
UpdateAttribute works if the problem is instead in QueryRecord eagerly
evaluating the queries.
Any help would be appreciated.
Cheers,
Mandeep
[1] 2018-11-13 14:46:24,899 ERROR [Timer-Driven Process Thread-1]
o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=0d5684e2-0167-1000-74c1-eb29a1401981] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to
java.lang.reflect.InvocationTargetException:
java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor916.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
Subject is empty
at
org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:66)
at
org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:315)
at
org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:203)
at
org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
at
org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:51)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:113)
at
org.apache.nifi.processors.standard.QueryRecord.setupQueues(QueryRecord.java:443)
... 14 common frames omitted
--
Mandeep Gill
nstack.com <http://www.nstack.com/> / +44 7961822575
Re: JsonPath expression language exception with QueryRecord processor
Posted by Otto Fowler <ot...@gmail.com>.
Can you create a JIRA with this information?
On November 13, 2018 at 10:16:36, Mandeep Gill (mandeep@nstack.com) wrote:
Hi there,
I'm hitting an issue using the `jsonPath` expression language function to
extract a query to use with the QueryRecord processor. The processor works
fine if the expression language subject is contained within the process
group variable registry, but fails upon starting with an
AttributeExpressionLanguageException if the subject was expected to to
exist within a flowfile attribute [1]
I've attached a template generated on NiFi 1.8.0 demonstrating the problem
- it only appears to be an issue with dynamic outputs from the QueryRecord
processors, as the same expression language statement works fine when used
as part of UpdateAttribute processor with the subject in a flowfile
attribute as per the template. I've dug into the codebase and can trace the
error to the evaluate function within the JsonPathEvaluator class, which
throws the exception if the variable can not be referenced. I have a local
fix at https://github.com/apache/nifi/compare/master...nstack:fix/jsonpath that
returns StringQueryResult("") if the subject is empty instead of throwing
the exception and this appears to work however I wonder that as
UpdateAttribute works if the problem is instead in QueryRecord eagerly
evaluating the queries.
Any help would be appreciated.
Cheers,
Mandeep
[1] 2018-11-13 14:46:24,899 ERROR [Timer-Driven Process Thread-1]
o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=0d5684e2-0167-1000-74c1-eb29a1401981] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to
java.lang.reflect.InvocationTargetException:
java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor916.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
Subject is empty
at
org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:66)
at
org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:315)
at
org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:203)
at
org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
at
org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:51)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:113)
at
org.apache.nifi.processors.standard.QueryRecord.setupQueues(QueryRecord.java:443)
... 14 common frames omitted
--
Mandeep Gill
nstack.com <http://www.nstack.com/> / +44 7961822575
Re: JsonPath expression language exception with QueryRecord processor
Posted by Mandeep Gill <ma...@nstack.com>.
Hi all,
Have created as an issue in Jira: NIFI-5817 (
https://issues.apache.org/jira/browse/NIFI-5817).
Thanks Mark, yes can confirm that adding `replaceNull` to the function
chain as described works around the issue - have updated the ticket to
reflect this.
Cheers,
Mandeep
On Tue, 13 Nov 2018 at 18:21 Mark Payne <ma...@hotmail.com> wrote:
> Hi Mandeep,
>
> I think this may actually be a bug in QueryRecord, in the way that it's
> handling the Expression Language.
> Do please file a JIRA for that. In the meantime, you can probably work
> around the issue using the replaceNull() method.
> So if your expression was ${myAttribute:jsonPath('/hello')} you could
> instead use ${myAttribute:replaceNull('{}'):jsonPath('/hello')}
>
> Thanks
> -Mark
>
> On Nov 13, 2018, at 10:16 AM, Mandeep Gill <ma...@nstack.com> wrote:
>
> Hi there,
>
> I'm hitting an issue using the `jsonPath` expression language function to
> extract a query to use with the QueryRecord processor. The processor works
> fine if the expression language subject is contained within the process
> group variable registry, but fails upon starting with an
> AttributeExpressionLanguageException if the subject was expected to to
> exist within a flowfile attribute [1]
>
> I've attached a template generated on NiFi 1.8.0 demonstrating the problem
> - it only appears to be an issue with dynamic outputs from the QueryRecord
> processors, as the same expression language statement works fine when used
> as part of UpdateAttribute processor with the subject in a flowfile
> attribute as per the template. I've dug into the codebase and can trace the
> error to the evaluate function within the JsonPathEvaluator class, which
> throws the exception if the variable can not be referenced. I have a local
> fix at https://github.com/apache/nifi/compare/master...nstack:fix/jsonpath that
> returns StringQueryResult("") if the subject is empty instead of throwing
> the exception and this appears to work however I wonder that as
> UpdateAttribute works if the problem is instead in QueryRecord eagerly
> evaluating the queries.
>
> Any help would be appreciated.
>
> Cheers,
> Mandeep
>
> [1] 2018-11-13 14:46:24,899 ERROR [Timer-Driven Process Thread-1]
> o.a.nifi.processors.standard.QueryRecord
> QueryRecord[id=0d5684e2-0167-1000-74c1-eb29a1401981] Failed to properly
> initialize Processor. If still scheduled to run, NiFi will attempt to
> initialize and run the Processor again after the 'Administrative Yield
> Duration' has elapsed. Failure is due to
> java.lang.reflect.InvocationTargetException:
> java.lang.reflect.InvocationTargetException
> java.lang.reflect.InvocationTargetException: null
> at sun.reflect.GeneratedMethodAccessor916.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
> at
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
> at
> org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by:
> org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
> Subject is empty
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:66)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:315)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:203)
> at
> org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
> at
> org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:51)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:113)
> at
> org.apache.nifi.processors.standard.QueryRecord.setupQueues(QueryRecord.java:443)
> ... 14 common frames omitted
>
> --
>
> Mandeep Gill
>
> nstack.com <http://www.nstack.com/> / +44 7961822575 <+44%207961%20822575>
>
> <QueryRecord_JsonPath_Issue.xml>
>
>
> --
Mandeep Gill
nstack.com <http://www.nstack.com/> / +44 7961822575
Re: JsonPath expression language exception with QueryRecord processor
Posted by Mark Payne <ma...@hotmail.com>.
Hi Mandeep,
I think this may actually be a bug in QueryRecord, in the way that it's handling the Expression Language.
Do please file a JIRA for that. In the meantime, you can probably work around the issue using the replaceNull() method.
So if your expression was ${myAttribute:jsonPath('/hello')} you could instead use ${myAttribute:replaceNull('{}'):jsonPath('/hello')}
Thanks
-Mark
On Nov 13, 2018, at 10:16 AM, Mandeep Gill <ma...@nstack.com>> wrote:
Hi there,
I'm hitting an issue using the `jsonPath` expression language function to extract a query to use with the QueryRecord processor. The processor works fine if the expression language subject is contained within the process group variable registry, but fails upon starting with an AttributeExpressionLanguageException if the subject was expected to to exist within a flowfile attribute [1]
I've attached a template generated on NiFi 1.8.0 demonstrating the problem - it only appears to be an issue with dynamic outputs from the QueryRecord processors, as the same expression language statement works fine when used as part of UpdateAttribute processor with the subject in a flowfile attribute as per the template. I've dug into the codebase and can trace the error to the evaluate function within the JsonPathEvaluator class, which throws the exception if the variable can not be referenced. I have a local fix at https://github.com/apache/nifi/compare/master...nstack:fix/jsonpath that returns StringQueryResult("") if the subject is empty instead of throwing the exception and this appears to work however I wonder that as UpdateAttribute works if the problem is instead in QueryRecord eagerly evaluating the queries.
Any help would be appreciated.
Cheers,
Mandeep
[1] 2018-11-13 14:46:24,899 ERROR [Timer-Driven Process Thread-1] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=0d5684e2-0167-1000-74c1-eb29a1401981] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.reflect.InvocationTargetException: java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor916.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException: Subject is empty
at org.apache.nifi.attribute.expression.language.evaluation.functions.JsonPathEvaluator.evaluate(JsonPathEvaluator.java:66)
at org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:315)
at org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:203)
at org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
at org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:51)
at org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
at org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
at org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:113)
at org.apache.nifi.processors.standard.QueryRecord.setupQueues(QueryRecord.java:443)
... 14 common frames omitted
--
Mandeep Gill
nstack.com<http://www.nstack.com/> / +44 7961822575
<QueryRecord_JsonPath_Issue.xml>