You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2019/03/04 19:50:06 UTC
Re: How to resolve classes for Groovy in ExecuteScript
Thank you all very much for your help. After adding a jar that did have the
LoggerFactory class, I still got logger factory errors. I suspect I may
have been pulling in the LoggerFactory class from a jar in lib that was in
some way either not compatible with Groovy, or not compatible with the
other java classes I had imported. I wish I had written down the error but
did not.
I felt there were too many challenges for me to solve in the time I had for
the task. For instance, I noticed using "jar tvf" that there were
LoggerFactory classes in several of the jars in lib (and if memory serves,
in lib_community too). Which one should I use with Groovy? Did it matter?
If I opted for the wrong jar could I be causing incompatibilities? I
couldn't say, and felt like I was no longer making progress.
Because there were so many uncertainties I opted to punt on that approach
and solve the problem using what is admittedly not as elegant an approach
as the Groovy script solution. I did this:
1. my main flow parses and logs to nifi-app.log. It runs between 0000 and
2100 daily via CRON timer driven configuration.
2. I adapted the solution found here
https://community.hortonworks.com/articles/65027/nifi-easy-custom-logging-of-diverse-sources-in-mer.html
to extract my parser json log statements from nifi-app.log to a custom log
file. Works great.
3. I set up a CRON timer driven flow that runs at 2130 each day, renaming
my custom log to customLog.log_[dt suffix], and re-establishes a zero byte
customLog.log to use fresh the next day.
4. I have a small final flow that is a List / Fetch running once daily at
2145, sending to bit bucket heaven any of my log backups aged more than ten
days.
Seems to be working without any issues so far. In cases where I *must*
safeguard against log management and main flow trying to battle for access
to the custom log, I use an empty semaphore file to ensure only the flow in
possession of the semaphore can run.
Thanks again for your help. I got close and intend to revisit when I have
more time. In the meantime, I hope more folks post similar solutions I
might learn from.
On Wed, Feb 27, 2019 at 7:15 PM Mike Thomsen <mi...@gmail.com> wrote:
> FYI, on 1.10.0-SNAPSHOT this works without modifying the classpath in
> ExecuteScript:
>
> import ch.qos.logback.classic.encoder.*
> import ch.qos.logback.core.rolling.*
>
> f = new RollingFileAppender()
> t = new ch.qos.logback.core.rolling.TimeBasedRollingPolicy()
>
> Also, see Shawn's point about you not importing LoggerFactory before
> referencing it.
>
> On Wed, Feb 27, 2019 at 11:06 AM Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
>> Where is your import for logger factory? If groovy can’t resolve a class
>> it then looks for a property of the same name which doesn’t exists.
>>
>> Shawn
>>
>> Sent from my iPhone
>>
>> On Feb 27, 2019, at 10:00 AM, James McMahon <js...@gmail.com> wrote:
>>
>> It is difficult to get this to work. Error shown below, as well as my
>> code. What needs to be done so that this works? I am willing to change from
>> Groovy to Python if there is a concrete example that works in Python you
>> can direct me to, for log rolling based on log size.
>>
>> What I did:
>> 1. established a basic Groovy script that did not try to employ a rolling
>> policy. That worked.
>> 2. found an example of a rolling log implementation, tried to bring that
>> in here. Got the "class not found" errors.
>> 3. Did a lot of digging to find libraries that offered the missing
>> classes. In some cases, they were already in nifi lib, in other cases I had
>> to download and install in lib_community
>> 4. Added import statements for the classes to my Groovy code
>> 5. In my ExecuteScript configuration, for Module Directory I set as:
>> /nifi/latest/lib,/nifi/latest/lib_community
>> 6. I am calling the Groovy Script from an ExecuteScript processor.
>>
>> The errors:
>> When I try to run the processor it now throws these two errors:
>> ExecuteScript[id=...] failed to process due to
>> javax.script.ScriptException: javax.script.ScriptException:
>> groovy.lang.MissingPropertyException: No such property: LoggerFactory for
>> class: ch.qos.logback.core.rolling.TimeBasedRollingPolicy
>> ExecuteScript[id=...] Failed to process session due to
>> javax.script.ScriptException: javax.script.ScriptException:
>> groovy.lang.MissingPropertyException: No such property: LoggerFactory for
>> class: ch.qos.logback.core.rolling.TimeBasedRollingPolicy
>>
>> The exceptions appear to be telling me that I can't get a LoggerFactory
>> for the TimeBasedRollingPolicy.
>> But if I can't get a LoggerFactory, then how should i make use of the
>> TimeBasedRollingPolicy?
>>
>> What I coded, pardon any typos from my hand jamming the code here for
>> your review:
>>
>> import org.apache.commons.io.IOUtils
>> import org.apache.log4j.*
>> import ch.qos.logback.classic.encoder.*
>> import ch.qos.logback.core.rolling.*
>> import java.nio.charset.*
>> def flowFile = session.get()
>> if (!flowFile) return
>> filename = customfilename.evaluateAttributeExpressions(flowFile).value
>> // filename is typically like this nifi-app.281719-3007.log
>>
>> f = new RollingFileAppender().with {
>> name = 'file appender'
>> file = "/opt/nifi/latest/logs/${filename}"
>> // the policy to roll the logs...
>> rollingPolicy = new TimeBasedRollingPolicy().with {
>> context = LoggerFactory.getILoggerFactory()
>> fileNamePattern =
>> "/opt/nifi/latest/logs/${filename}.%date{yyyy-mm-dd}.%i.log"
>> maxHistory = 10
>> //timeBasedFileNamingAndTriggerPolicy = new
>> SizeAndTimeBasedFNATP().with{
>> timeBasedFileNamingAndTriggerPolicy = new
>> SizeAndTimeBasedRolingPolicy().with{
>> context = LoggerFactory.getILoggerFactory()
>> maxFileSise = '30MB'
>> }
>> }
>> context = LoggerFactory.getILoggerFactory()
>> encoder = new PatternLayoutEncoder().with {
>> context = LoggerFactory.getILoggerFactory()
>> pattern = "%date{HH:mm:ss.SSS} [%thread] %-5level %logger{35} -
>> %msg%n"
>> start()
>> it
>> }
>> log.addAppender(it)
>> }
>> flowFile = session.write(flowFile, {inputStream, outputStream ->
>> try {
>> f.append(IOUtils.toString(inputStream,
>> standardCharsets.UTF_8).trim().replaceAll("value: ","")+'\n')
>> }
>> catch(e) {
>> log.error("Error during processing to custom log: ${filename}",
>> e)
>> }
>> } as StreamCallback)
>> session.transfer(flowFile, REL_SUCCESS)
>>
>> On Tue, Feb 26, 2019 at 4:10 PM Matt Burgess <ma...@apache.org>
>> wrote:
>>
>>> Oh right, not sure why I thought Jim was using ExecuteGroovyScript.
>>> For ExecuteScript the stuff in lib/ should be available, if it's not
>>> then I did something wrong :P
>>>
>>> On Tue, Feb 26, 2019 at 4:04 PM Bryan Bende <bb...@gmail.com> wrote:
>>> >
>>> > Ah I didn't realize we also had ExecuteGroovyScript.
>>> >
>>> > That processor has a different property called "Additional Classpath"
>>> > and it looks like it ends up using GroovyClassLoader with the parent
>>> > set to whatever loaded GroovyShell.class, which in this case I would
>>> > assume is the NAR class loader for the nifi-groovyx-nar.
>>> >
>>> > Not sure which processor was being used here.
>>> >
>>> > On Tue, Feb 26, 2019 at 3:54 PM Matt Burgess <ma...@apache.org>
>>> wrote:
>>> > >
>>> > > Yeah I was surprised too for things like PatternLayoutEncoder which
>>> is
>>> > > in logback-classic, but for TimeBasedRollingPolicy and such you might
>>> > > need to bring in apache-log4j-extras. I didn't write
>>> > > ExecuteGroovyScript so I'm not sure if there's something different
>>> > > about the classloader, I'd have to dig into the code.
>>> > >
>>> > > On Tue, Feb 26, 2019 at 3:26 PM Bryan Bende <bb...@gmail.com>
>>> wrote:
>>> > > >
>>> > > > Hello,
>>> > > >
>>> > > > Have you tried setting the "Modules Directory" of ExecuteScript to
>>> a
>>> > > > directory that has the corresponding JARs you need?
>>> > > >
>>> > > > Matt B is probably more familiar with how the class loader works
>>> for
>>> > > > ExecuteScript, but I would expect some of these to be available in
>>> > > > root class loader since NiFi has slf4j and logback in the lib dir.
>>> > > >
>>> > > > jcl-over-slf4j-1.7.25.jar
>>> > > > jul-to-slf4j-1.7.25.jar
>>> > > > log4j-over-slf4j-1.7.25.jar
>>> > > > logback-classic-1.2.3.jar
>>> > > > logback-core-1.2.3.jar
>>> > > > slf4j-api-1.7.25.jar
>>> > > >
>>> > > > -Bryan
>>> > > >
>>> > > >
>>> > > > On Tue, Feb 26, 2019 at 2:47 PM James McMahon <
>>> jsmcmahon3@gmail.com> wrote:
>>> > > > >
>>> > > > > I am attempting to establish a rolling log from a Groovy script
>>> executed in an ExecuteScript processor. am new to Groovy. I used yum
>>> install to install Groovy on my CentOS 7 system. I'm running with openjdk
>>> 1.8.9_191. When I try to use the following classes in my script,
>>> ExecuteScript throws an error saying they cannot be resolved:
>>> > > > >
>>> > > > > SizeAndTimeBasedFNATP
>>> > > > > TimeBasedRollingPolicy
>>> > > > > PatternLayoutEncoder
>>> > > > > RollingFileAppender
>>> > > > >
>>> > > > > Can anyone tell me how to fix this problem? Do i need to install
>>> special libraries - and if so, in groovy, or in java?
>>> > > > >
>>> > > > > I was referred to this solution and its Groovy script, and am
>>> using it as the basis for my log redirection:
>>> > > > >
>>> https://community.hortonworks.com/articles/65027/nifi-easy-custom-logging-of-diverse-sources-in-mer.html
>>> > > > > It works great with a standard non-rotating log.
>>> > > > >
>>> > > > > This is the solution I've adapted from StackOverflow in an
>>> effort to establish a rolling log file handle:
>>> > > > >
>>> https://stackoverflow.com/questions/24436239/groovy-slf4j-rotation-and-backup-log-files
>>> > > > > But truth is I'm grasping in the dark, not understanding how to
>>> get these libraries installed and made known to ExecuteScript.
>>> > > > >
>>> > > > > Thank you.
>>> > > > > Jim
>>>
>>