You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chris Herssens <ch...@gmail.com> on 2020/05/11 07:46:36 UTC
groovy script in nifi 1.11 : unable to loasd FastStringService
Hello,
I try to implement a groovy script where I'm using jsonOutput.
With nifi 1.5 the script works, but If I'm try to use the same groovy
script with nifi 1.11.4, I get
"unable to load FastStringService"
example code :
class GroovyRecordSetWriter implements RecordSetWriter {
....
@Override
WriteResult write(Record r) throws IOException {
...
def j = JsonOutput.toJson([name: 'John Doe', age: 42])
out.write(j.getBytes())
...
Regards,
Chris
Re: groovy script in nifi 1.11 : unable to loasd FastStringService
Posted by Mark Payne <ma...@hotmail.com>.
Matt, Chris, et al.
I thought I had replied to this thread already noting that I created NIFI-7447 [1] to look at how to address this at the framework level.
I just wanted to close the loop on this, though. This issue was very easy to replicate in 1.11.4. I tested this scenario with the fix for NIFI-7447 and I tested many other scenarios around Record processors, schema registry, scripted and unscripted record writers, as well as some issues around JMS processors that had crept up before. The fix for NIFI-7447 has now been merged to master, and I believe it addresses all of these concerns.
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7447
On May 11, 2020, at 9:27 PM, Matt Burgess <ma...@apache.org>> wrote:
(Moved users to BCC, added dev as I'm about to get into the weeds :)
So this isn't a Groovy/scripting issue per se, but I'm not sure if
anything outside the scripted controller services are affected
currently. In general for controller service (CS) implementations, we
have a proxy called StandardControllerServiceInvocationHandler that
gets called when we want to invoke a method on the CS implementation.
That handler ensures the methods are called using the controller
service's NAR classloader:
try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(extensionManager,
originalService.getClass(), originalService.getIdentifier())) {
return method.invoke(originalService, args);
} ...
In this case the controller service interface is
RecordSetWriterFactory, not RecordSetWriter. So when createWriter() is
called on the ScriptedRecordSetWriter, it delegates to the script's
defined RecordSetWriterFactory, with the NAR classloader as the
thread's context classloader, and all is well. But when a class like
ConvertRecord calls createWriter() to get a RecordSetWriter, it gets a
reference to such but does not proxy the calls to RecordSetWriter
methods (as RecordSetWriter does not inherit from ControllerService),
which means it is called with the processor's thread's classloader,
not the controller service's classloader. This wouldn't normally cause
a problem unless something inside the RecordReader/RecordSetWriter
implementation uses the thread's context classloader to load classes.
In the case of Groovy, as of 2.5.x (which we've since upgraded to),
it uses ServiceLoader (with the thread's context classloader) to find
a FastStringService implementation when using JsonOutput. But since
the processor (ConvertRecord in this case) doesn't have groovy-json in
its dependencies, the script doesn't find the class and thus the error
occurs.
That's a mouthful :) Also I'm not quite sure what to do about it. A
workaround may be to pass the thread's context class loader from
within the scripted RecordSetWriterFactory implementation to the
scripted RecordSetWriter implementation, but then we'd need something
akin to the code block above to set the context classloader in each
method, do the work, then restore (for cleanliness). A slightly
quicker way may be to only implement that in your script when you're
dealing with one of these pesky classes like
JsonOutput/FastServiceString, so perhaps if you're lucky only in one
method.
We might also look at wrapping calls to RecordReader and
RecordSetWriter methods in something akin to the above, knowing that
they are coupled to the CS factory interfaces as such. But without the
framework creating a proxy, we may need to expose the information
about the factory's classloader to processors using the "derived"
classes. This increases the coupling between the factories and the
readers/writers but since they are pretty much part of an "ecosystem"
that might be a viable option. Definitely interested in any thoughts
about how to proceed (I'm looking at you Payne lol).
Regards,
Matt
On Mon, May 11, 2020 at 6:34 PM Matt Burgess <ma...@apache.org>> wrote:
Chris,
There's definitely something funky going on there, the script doesn't
get the same classloader chain that the ScriptedRecordSetWriter (that
loads the script) does, instead it gets one with the standard NAR as
the parent instead of the scripting NAR. I'm looking into it now.
BTW for scripted component issues, you might be better off emailing
the dev list, there may be more folks in there familiar with the NiFi
code and scripting languages and such. Having said that, we can
maintain this thread until we get to the bottom of the issue.
Regards,
Matt
On Mon, May 11, 2020 at 3:47 AM Chris Herssens <ch...@gmail.com>> wrote:
Hello,
I try to implement a groovy script where I'm using jsonOutput.
With nifi 1.5 the script works, but If I'm try to use the same groovy script with nifi 1.11.4, I get
"unable to load FastStringService"
example code :
class GroovyRecordSetWriter implements RecordSetWriter {
....
@Override
WriteResult write(Record r) throws IOException {
...
def j = JsonOutput.toJson([name: 'John Doe', age: 42])
out.write(j.getBytes())
...
Regards,
Chris
Re: groovy script in nifi 1.11 : unable to loasd FastStringService
Posted by Matt Burgess <ma...@apache.org>.
(Moved users to BCC, added dev as I'm about to get into the weeds :)
So this isn't a Groovy/scripting issue per se, but I'm not sure if
anything outside the scripted controller services are affected
currently. In general for controller service (CS) implementations, we
have a proxy called StandardControllerServiceInvocationHandler that
gets called when we want to invoke a method on the CS implementation.
That handler ensures the methods are called using the controller
service's NAR classloader:
try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(extensionManager,
originalService.getClass(), originalService.getIdentifier())) {
return method.invoke(originalService, args);
} ...
In this case the controller service interface is
RecordSetWriterFactory, not RecordSetWriter. So when createWriter() is
called on the ScriptedRecordSetWriter, it delegates to the script's
defined RecordSetWriterFactory, with the NAR classloader as the
thread's context classloader, and all is well. But when a class like
ConvertRecord calls createWriter() to get a RecordSetWriter, it gets a
reference to such but does not proxy the calls to RecordSetWriter
methods (as RecordSetWriter does not inherit from ControllerService),
which means it is called with the processor's thread's classloader,
not the controller service's classloader. This wouldn't normally cause
a problem unless something inside the RecordReader/RecordSetWriter
implementation uses the thread's context classloader to load classes.
In the case of Groovy, as of 2.5.x (which we've since upgraded to),
it uses ServiceLoader (with the thread's context classloader) to find
a FastStringService implementation when using JsonOutput. But since
the processor (ConvertRecord in this case) doesn't have groovy-json in
its dependencies, the script doesn't find the class and thus the error
occurs.
That's a mouthful :) Also I'm not quite sure what to do about it. A
workaround may be to pass the thread's context class loader from
within the scripted RecordSetWriterFactory implementation to the
scripted RecordSetWriter implementation, but then we'd need something
akin to the code block above to set the context classloader in each
method, do the work, then restore (for cleanliness). A slightly
quicker way may be to only implement that in your script when you're
dealing with one of these pesky classes like
JsonOutput/FastServiceString, so perhaps if you're lucky only in one
method.
We might also look at wrapping calls to RecordReader and
RecordSetWriter methods in something akin to the above, knowing that
they are coupled to the CS factory interfaces as such. But without the
framework creating a proxy, we may need to expose the information
about the factory's classloader to processors using the "derived"
classes. This increases the coupling between the factories and the
readers/writers but since they are pretty much part of an "ecosystem"
that might be a viable option. Definitely interested in any thoughts
about how to proceed (I'm looking at you Payne lol).
Regards,
Matt
On Mon, May 11, 2020 at 6:34 PM Matt Burgess <ma...@apache.org> wrote:
>
> Chris,
>
> There's definitely something funky going on there, the script doesn't
> get the same classloader chain that the ScriptedRecordSetWriter (that
> loads the script) does, instead it gets one with the standard NAR as
> the parent instead of the scripting NAR. I'm looking into it now.
>
> BTW for scripted component issues, you might be better off emailing
> the dev list, there may be more folks in there familiar with the NiFi
> code and scripting languages and such. Having said that, we can
> maintain this thread until we get to the bottom of the issue.
>
> Regards,
> Matt
>
> On Mon, May 11, 2020 at 3:47 AM Chris Herssens <ch...@gmail.com> wrote:
> >
> > Hello,
> >
> > I try to implement a groovy script where I'm using jsonOutput.
> > With nifi 1.5 the script works, but If I'm try to use the same groovy script with nifi 1.11.4, I get
> > "unable to load FastStringService"
> >
> > example code :
> >
> > class GroovyRecordSetWriter implements RecordSetWriter {
> > ....
> > @Override
> > WriteResult write(Record r) throws IOException {
> > ...
> > def j = JsonOutput.toJson([name: 'John Doe', age: 42])
> > out.write(j.getBytes())
> >
> > ...
> >
> > Regards,
> > Chris
Re: groovy script in nifi 1.11 : unable to loasd FastStringService
Posted by Matt Burgess <ma...@apache.org>.
(Moved users to BCC, added dev as I'm about to get into the weeds :)
So this isn't a Groovy/scripting issue per se, but I'm not sure if
anything outside the scripted controller services are affected
currently. In general for controller service (CS) implementations, we
have a proxy called StandardControllerServiceInvocationHandler that
gets called when we want to invoke a method on the CS implementation.
That handler ensures the methods are called using the controller
service's NAR classloader:
try (final NarCloseable narCloseable =
NarCloseable.withComponentNarLoader(extensionManager,
originalService.getClass(), originalService.getIdentifier())) {
return method.invoke(originalService, args);
} ...
In this case the controller service interface is
RecordSetWriterFactory, not RecordSetWriter. So when createWriter() is
called on the ScriptedRecordSetWriter, it delegates to the script's
defined RecordSetWriterFactory, with the NAR classloader as the
thread's context classloader, and all is well. But when a class like
ConvertRecord calls createWriter() to get a RecordSetWriter, it gets a
reference to such but does not proxy the calls to RecordSetWriter
methods (as RecordSetWriter does not inherit from ControllerService),
which means it is called with the processor's thread's classloader,
not the controller service's classloader. This wouldn't normally cause
a problem unless something inside the RecordReader/RecordSetWriter
implementation uses the thread's context classloader to load classes.
In the case of Groovy, as of 2.5.x (which we've since upgraded to),
it uses ServiceLoader (with the thread's context classloader) to find
a FastStringService implementation when using JsonOutput. But since
the processor (ConvertRecord in this case) doesn't have groovy-json in
its dependencies, the script doesn't find the class and thus the error
occurs.
That's a mouthful :) Also I'm not quite sure what to do about it. A
workaround may be to pass the thread's context class loader from
within the scripted RecordSetWriterFactory implementation to the
scripted RecordSetWriter implementation, but then we'd need something
akin to the code block above to set the context classloader in each
method, do the work, then restore (for cleanliness). A slightly
quicker way may be to only implement that in your script when you're
dealing with one of these pesky classes like
JsonOutput/FastServiceString, so perhaps if you're lucky only in one
method.
We might also look at wrapping calls to RecordReader and
RecordSetWriter methods in something akin to the above, knowing that
they are coupled to the CS factory interfaces as such. But without the
framework creating a proxy, we may need to expose the information
about the factory's classloader to processors using the "derived"
classes. This increases the coupling between the factories and the
readers/writers but since they are pretty much part of an "ecosystem"
that might be a viable option. Definitely interested in any thoughts
about how to proceed (I'm looking at you Payne lol).
Regards,
Matt
On Mon, May 11, 2020 at 6:34 PM Matt Burgess <ma...@apache.org> wrote:
>
> Chris,
>
> There's definitely something funky going on there, the script doesn't
> get the same classloader chain that the ScriptedRecordSetWriter (that
> loads the script) does, instead it gets one with the standard NAR as
> the parent instead of the scripting NAR. I'm looking into it now.
>
> BTW for scripted component issues, you might be better off emailing
> the dev list, there may be more folks in there familiar with the NiFi
> code and scripting languages and such. Having said that, we can
> maintain this thread until we get to the bottom of the issue.
>
> Regards,
> Matt
>
> On Mon, May 11, 2020 at 3:47 AM Chris Herssens <ch...@gmail.com> wrote:
> >
> > Hello,
> >
> > I try to implement a groovy script where I'm using jsonOutput.
> > With nifi 1.5 the script works, but If I'm try to use the same groovy script with nifi 1.11.4, I get
> > "unable to load FastStringService"
> >
> > example code :
> >
> > class GroovyRecordSetWriter implements RecordSetWriter {
> > ....
> > @Override
> > WriteResult write(Record r) throws IOException {
> > ...
> > def j = JsonOutput.toJson([name: 'John Doe', age: 42])
> > out.write(j.getBytes())
> >
> > ...
> >
> > Regards,
> > Chris
Re: groovy script in nifi 1.11 : unable to loasd FastStringService
Posted by Matt Burgess <ma...@apache.org>.
Chris,
There's definitely something funky going on there, the script doesn't
get the same classloader chain that the ScriptedRecordSetWriter (that
loads the script) does, instead it gets one with the standard NAR as
the parent instead of the scripting NAR. I'm looking into it now.
BTW for scripted component issues, you might be better off emailing
the dev list, there may be more folks in there familiar with the NiFi
code and scripting languages and such. Having said that, we can
maintain this thread until we get to the bottom of the issue.
Regards,
Matt
On Mon, May 11, 2020 at 3:47 AM Chris Herssens <ch...@gmail.com> wrote:
>
> Hello,
>
> I try to implement a groovy script where I'm using jsonOutput.
> With nifi 1.5 the script works, but If I'm try to use the same groovy script with nifi 1.11.4, I get
> "unable to load FastStringService"
>
> example code :
>
> class GroovyRecordSetWriter implements RecordSetWriter {
> ....
> @Override
> WriteResult write(Record r) throws IOException {
> ...
> def j = JsonOutput.toJson([name: 'John Doe', age: 42])
> out.write(j.getBytes())
>
> ...
>
> Regards,
> Chris