You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2019/02/21 20:20:38 UTC
Unable to write to rotating log file from ExecuteScript
We are running NiFi 1.8, trying to log to a rotating log file from a python
script executed by an ExecuteScript processor. We are seeing no output to
the log file.
I seem to be establishing the log file handler and the logger without any
errors.
I see the log file in my log output directory, so I know it gets created.
Can this approach be used from Execute Script? What am i neglecting?
My approach is as follows:
flowFiles = session.get(100)
for flowFile in flowFiles :
if (flowFile != None) :
fh =
logging.handlers.RotatingFileHandler('/opt/nifi/logs/myLog.log',
maxBytes=500000, backupCount=20)
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s
%(message)s')
fh.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(fh)
incoming = flowFile.getAttribute('filename)
logger.info('about to process: %s', incoming)
flowFile = session.write(flowFile, PyStreamCallback()) # heavy
lifting in this PyStreamCallback
logger.info('completed processing: %s', incoming)
fh.close()
while logger.handlers :
logger.handlers.pop()
del fh
session.transfer(flowFile, REL_SUCCESS)
Thanks in advance for your help. -Jim
Re: Unable to write to rotating log file from ExecuteScript
Posted by James McMahon <js...@gmail.com>.
Andy, thank you very much. This is a section of my code excerpted and hand
jammed in here by me to give the question context. I apologize: I
introduced those typos when hand jamming. They are not in the code, but
thanks for pointing them out here. I do have the correct imports.
Unfortunately I have to depend on a PyStreamCallback() class, and so can't
employ a direct api call such as WriteContentCallback("Hello there this is
my data”) in my code.
From what I am seeing, logging to a rotating log file from an ExecuteScript
results in odd multiple writes to the log that increase in number with each
subsequent flowfile. I suspect it has to do with the way the log handle is
not closed properly.
I gather from some research that the way to do this is to employ an
InvokeScriptedProcessor (ISP) - not ExecuteScript. Problem is, I can't find
an example that shows how the PyStreamCallback is integrated with the
processor class in an ISP. Matt Burgess speaks to it here
http://funnifi.blogspot.com/2017/11/invokescriptedprocessor-template.html
but this example doesn't show how to integrate the PyStreamCallback. It is
typically a class of its own in the python ExecuteScript python script, and
I don't see how I can fold that into the def executeScript() in this
section of Matt's example:
class E():
def __init__(self):
pass
def executeScript(self,session, context, log, REL_SUCCESS, REL_FAILURE):
#////////////////////////////////////////////////////////////
#// Replace 'pass' with your code
#////////////////////////////////////////////////////////////
pass
#end class
Do you by chance know of a more concrete example for Python,
InvokeScriptedProcessor, rotating log, PyStreamCallback included? Perhaps
even just offer a simple example of the way the execute script and the
callback would be structured in the executeScript method shown above?
Jim
On Thu, Feb 21, 2019 at 9:52 PM Andy LoPresto <al...@apache.org> wrote:
> Jim,
>
> I had started to provide advice on configuring the logback.xml file when I
> realized you want to output log messages directly to your own log file, not
> using the provided log interface that ExecuteScript offers. I don’t know
> Python well enough to be sure, but I get an error trying to execute your
> script because “logging.handlers.RotatingFileHandler…” can’t be found. You
> may need to configure your processor with a specific module in the modules
> directory, or import that class explicitly. I also want to note there is an
> unclosed single quote on line 11 [ incoming =
> flowFile.getAttribute(‘filename) ] which may be causing some problems — I
> had to close it before I could detect the actual issue. I also had to
> change line 13 to [ flowFile = session.write(flowFile,
> WriteContentCallback("Hello there this is my data”)) ] in order to run this
> example locally.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Feb 21, 2019, at 12:20 PM, James McMahon <js...@gmail.com> wrote:
>
> We are running NiFi 1.8, trying to log to a rotating log file from a
> python script executed by an ExecuteScript processor. We are seeing no
> output to the log file.
> I seem to be establishing the log file handler and the logger without any
> errors.
> I see the log file in my log output directory, so I know it gets created.
> Can this approach be used from Execute Script? What am i neglecting?
>
> My approach is as follows:
>
> flowFiles = session.get(100)
> for flowFile in flowFiles :
> if (flowFile != None) :
> fh =
> logging.handlers.RotatingFileHandler('/opt/nifi/logs/myLog.log',
> maxBytes=500000, backupCount=20)
> fh.setLevel(logging.DEBUG)
> formatter = logging.Formatter('%(asctime)s %(name)s
> %(levelname)s %(message)s')
> fh.setFormatter(formatter)
> logger = logging.getLogger()
> logger.addHandler(fh)
>
> incoming = flowFile.getAttribute('filename)
> logger.info('about to process: %s', incoming)
> flowFile = session.write(flowFile, PyStreamCallback()) # heavy
> lifting in this PyStreamCallback
> logger.info('completed processing: %s', incoming)
>
> fh.close()
> while logger.handlers :
> logger.handlers.pop()
> del fh
>
> session.transfer(flowFile, REL_SUCCESS)
>
> Thanks in advance for your help. -Jim
>
>
>
Re: Unable to write to rotating log file from ExecuteScript
Posted by Andy LoPresto <al...@apache.org>.
Jim,
I had started to provide advice on configuring the logback.xml file when I realized you want to output log messages directly to your own log file, not using the provided log interface that ExecuteScript offers. I don’t know Python well enough to be sure, but I get an error trying to execute your script because “logging.handlers.RotatingFileHandler…” can’t be found. You may need to configure your processor with a specific module in the modules directory, or import that class explicitly. I also want to note there is an unclosed single quote on line 11 [ incoming = flowFile.getAttribute(‘filename) ] which may be causing some problems — I had to close it before I could detect the actual issue. I also had to change line 13 to [ flowFile = session.write(flowFile, WriteContentCallback("Hello there this is my data”)) ] in order to run this example locally.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 21, 2019, at 12:20 PM, James McMahon <js...@gmail.com> wrote:
>
> We are running NiFi 1.8, trying to log to a rotating log file from a python script executed by an ExecuteScript processor. We are seeing no output to the log file.
> I seem to be establishing the log file handler and the logger without any errors.
> I see the log file in my log output directory, so I know it gets created.
> Can this approach be used from Execute Script? What am i neglecting?
>
> My approach is as follows:
>
> flowFiles = session.get(100)
> for flowFile in flowFiles :
> if (flowFile != None) :
> fh = logging.handlers.RotatingFileHandler('/opt/nifi/logs/myLog.log', maxBytes=500000, backupCount=20)
> fh.setLevel(logging.DEBUG)
> formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s %(message)s')
> fh.setFormatter(formatter)
> logger = logging.getLogger()
> logger.addHandler(fh)
>
> incoming = flowFile.getAttribute('filename)
> logger.info <http://logger.info/>('about to process: %s', incoming)
> flowFile = session.write(flowFile, PyStreamCallback()) # heavy lifting in this PyStreamCallback
> logger.info <http://logger.info/>('completed processing: %s', incoming)
>
> fh.close()
> while logger.handlers :
> logger.handlers.pop()
> del fh
>
> session.transfer(flowFile, REL_SUCCESS)
>
> Thanks in advance for your help. -Jim