You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Mark Petronic <ma...@gmail.com> on 2015/10/31 17:35:08 UTC
LogAttribute - Sending that output to a custom logger?
>From the code, it appears it cannot be done as the attribute logging
goes the same getLogger() instance as the normal nifi-app traces. Has
anyone considered making that configurable, maybe allowing you do
define a different logger name for LogAttribute then creating that
logger definition in log back conf allowing flexibility? I'm using
attribute logging heavily as I try to better learn/debug Nifi (it give
you a nice 'under the hood' view of the flow) and build up some flows
and feel it would be beneficial to be able to capture the LogAttribte
message by themselves for more clarity on what is happening. I would
not mind maybe trying to implement this feature as my first crack at
contributing to the project. Seems like a fairly easy one that would
allow me to "go through the motions" of a full pull request process
and iron out the process. Anyone have any thoughts on this?
Re: LogAttribute - Sending that output to a custom logger?
Posted by Adam Taft <ad...@adamtaft.com>.
This thread has forked into two different conversations: 1. improvements
to LogAttribute processor; 2. improvements to processor documentation.
1) re: improvements to LogAttribute - we already have NIFI-67 [1] that
suggests a number of improvements to LogAttribute. One of these is the use
of a custom name for the logger so that logback rules can be written
against that name.
While the provenance engine is great for many scenarios, in my opinion, it
doesn't replace the need for true text-based logging. The tooling for log
processing is very mature and there's no ability to "grep" a provenance
repository, migrate or offload provenance logs into deep storage, store log
events into a database, or do any other cool syslogd or logback type
things. Being able to capture and log a flowfile at the exact right place
in the data flow and processing it using the command line is an extremely
valuable tool in the toolkit.
For a long time, I've wanted to work on at least some of the things
mentioned in NIFI-67 and will hopefully get to do so time willing. Having
a custom "name" for the LogAttribute processor seems like a no-brainer.
Contributions for this should definitely be welcome!
2) improvements to processor document - I agree, even as a somewhat
seasoned NIFI user, I still have a hard time reading and understanding the
processor documentation. I often do exactly what Mark P. suggests and
instead go directly to the source. Any contribution towards better
processor documentation is greatly appreciated!
[1] https://issues.apache.org/jira/browse/NIFI-67
On Mon, Nov 2, 2015 at 1:54 PM, Aldrin Piri <al...@gmail.com> wrote:
> We greatly appreciate contributions. Your prescribed approach sounds great
> and if you are willing to give us a few cycles pointing out, and optionally
> correcting, the items that are in need of improvement, we will certainly
> incorporate.
>
> Thanks!
>
> On Mon, Nov 2, 2015 at 1:28 PM, Mark Petronic <ma...@gmail.com>
> wrote:
>
> > I'm sort of in the camp of "don't come with a complaint if you don't come
> > with a solution" and hesitated to even raise the documentation comment
> > without just fixing it myself. How about this, I just do some updates on
> > some processor docs myself and use that as my first contribution to work
> > through the process of committing to this project?
> >
> > But, to give you one quick example, EvaluateJSONPath (which, btw has
> pretty
> > good docs otherwise) does not mention HOW to extract the JSON you are
> > interested in. I had to look at the code to figure out it used this:
> > https://github.com/jayway/JsonPath. Ok, that was not hard, I admit, but,
> > as
> > a user, should I need to look at the code for such information? I submit,
> > no. Me personally, I like to dig into the code. So, this is more a
> comment
> > on "overall goodness" for the general new user experience.
> >
> > I agree with your assessment of 'new user vibe' as I am starting to not
> > notice it as much. lol
> >
> > On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <jo...@gmail.com> wrote:
> >
> > > Mark
> > >
> > > All fair points. Can you please point out which processor docs
> > > specifically should be better. Let's fix em..you will quickly lose
> that
> > > new user vibe and not notice what needs to improve as much. We need to
> > > make the new user experience awesome.
> > >
> > > Thanks
> > > Joe
> > > On Nov 2, 2015 10:08 AM, "Mark Petronic" <ma...@gmail.com>
> wrote:
> > >
> > > > My primary use is for understanding Nifi. I like to direct various
> > > > processors output into both their logical next processor stage as
> well
> > as
> > > > into a log attribute processor. Then I tail the Nifi app log file and
> > > watch
> > > > what happens - in real time. I do not intend to use this for long
> term
> > > log
> > > > retention. I agree that providence is the right choice for that. So,
> > the
> > > > only reason I wanted to allow configuration of a custom logger was
> > simply
> > > > to isolate all the attribute-rich logging from the normal logging
> > > because I
> > > > was primarily interested in the attribute flows as a way to (a)
> better
> > > > understand what a processor emits because, frankly, the documentation
> > of
> > > > some of the processors is very sparse. So, I learn imperatively, so
> to
> > > > speak. I say that as a new user. I feel I should be able to get a
> > pretty
> > > > good understanding of a processor by reading the usage. But I am
> > finding
> > > > that the documentation, in some cases, is more like what I like to
> > refer
> > > to
> > > > as, "note to self" documentation. Great if you are the guy who wrote
> > the
> > > > processor with those "insights" - not so great if you are not the
> > > > developer. So, then I need to dig up the code. That should not be
> > needed
> > > as
> > > > the first step of understanding a processor as a new user. There is
> > some
> > > > well documented processors but not all are, IMHO. (b) Validate my
> flows
> > > > with some test data and verify attribute values look correct and
> > routing
> > > is
> > > > happen on them as expected, etc. Again, easier, IMO, to see in the
> logs
> > > > than digging into the providence data.
> > > >
> > > > Maybe this is just a good "private" feature for me so maybe I will
> just
> > > > create a private version to use on my own. I already have it working
> > but
> > > > would need more polish to achieve PR status. Maybe this is the sort
> of
> > > > thing that others would not find beneficial? That's fine. There are
> > > others
> > > > ways I can contribute in the future. I'm still having fun! :)
> > > >
> > > > On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <jo...@gmail.com>
> wrote:
> > > >
> > > > > Mark Petronic,
> > > > >
> > > > > I share Payne's perspective on this. But I'd also like to work
> with
> > > > > you to better understand the workflow. For those of us that have
> > used
> > > > > this tool for a long time there is a lot we take for granted from a
> > > > > new user perspective. We believe the provenance feature to
> provide a
> > > > > far superior option to understanding how an item went through the
> > > > > system and the timing and what we knew when and so on. But, it
> would
> > > > > be great to understand it from your perspective as someone learning
> > > > > NiFi. Not meaning to take away from your proposed contrib - that
> > > > > would be great too. Just want to see if the prov user experience
> > > > > solves what you're looking for and if not can we make it do that.
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <ma...@hotmail.com>
> > > > wrote:
> > > > > > Mark,
> > > > > >
> > > > > > To make sure that I understand what you're proposing, you want to
> > > add a
> > > > > property to
> > > > > > LogAttribute that allows users to provide a custom logger name?
> > > > > >
> > > > > > If that is indeed what you are suggesting then I think it's a
> great
> > > > idea.
> > > > > >
> > > > > > That being said, in practice I rarely ever use LogAttribute and
> we
> > > even
> > > > > considered removing
> > > > > > it from the codebase before we open sourced, because the Data
> > > > Provenance
> > > > > provides a
> > > > > > much better view of what's going on to debug your flows.
> > > > > >
> > > > > > I know you're pretty new to NiFi, so if you've not yet had a
> chance
> > > to
> > > > > play with the Provenance,
> > > > > > you can see the section in the User Guide at
> > > > >
> > > >
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > > > <
> > > > >
> > > >
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > > > >
> > > > > >
> > > > > > If you're interested in updating the LogAttribute processor,
> > though,
> > > > > we'd be happy to have
> > > > > > that contribution added, as it does make the Processor more
> usable.
> > > > > >
> > > > > > Thanks
> > > > > > -Mark
> > > > > >
> > > > > >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <
> > markpetronic@gmail.com
> > > >
> > > > > wrote:
> > > > > >>
> > > > > >> From the code, it appears it cannot be done as the attribute
> > logging
> > > > > >> goes the same getLogger() instance as the normal nifi-app
> traces.
> > > Has
> > > > > >> anyone considered making that configurable, maybe allowing you
> do
> > > > > >> define a different logger name for LogAttribute then creating
> that
> > > > > >> logger definition in log back conf allowing flexibility? I'm
> using
> > > > > >> attribute logging heavily as I try to better learn/debug Nifi
> (it
> > > give
> > > > > >> you a nice 'under the hood' view of the flow) and build up some
> > > flows
> > > > > >> and feel it would be beneficial to be able to capture the
> > > LogAttribte
> > > > > >> message by themselves for more clarity on what is happening. I
> > would
> > > > > >> not mind maybe trying to implement this feature as my first
> crack
> > at
> > > > > >> contributing to the project. Seems like a fairly easy one that
> > would
> > > > > >> allow me to "go through the motions" of a full pull request
> > process
> > > > > >> and iron out the process. Anyone have any thoughts on this?
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: LogAttribute - Sending that output to a custom logger?
Posted by Aldrin Piri <al...@gmail.com>.
We greatly appreciate contributions. Your prescribed approach sounds great
and if you are willing to give us a few cycles pointing out, and optionally
correcting, the items that are in need of improvement, we will certainly
incorporate.
Thanks!
On Mon, Nov 2, 2015 at 1:28 PM, Mark Petronic <ma...@gmail.com>
wrote:
> I'm sort of in the camp of "don't come with a complaint if you don't come
> with a solution" and hesitated to even raise the documentation comment
> without just fixing it myself. How about this, I just do some updates on
> some processor docs myself and use that as my first contribution to work
> through the process of committing to this project?
>
> But, to give you one quick example, EvaluateJSONPath (which, btw has pretty
> good docs otherwise) does not mention HOW to extract the JSON you are
> interested in. I had to look at the code to figure out it used this:
> https://github.com/jayway/JsonPath. Ok, that was not hard, I admit, but,
> as
> a user, should I need to look at the code for such information? I submit,
> no. Me personally, I like to dig into the code. So, this is more a comment
> on "overall goodness" for the general new user experience.
>
> I agree with your assessment of 'new user vibe' as I am starting to not
> notice it as much. lol
>
> On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <jo...@gmail.com> wrote:
>
> > Mark
> >
> > All fair points. Can you please point out which processor docs
> > specifically should be better. Let's fix em..you will quickly lose that
> > new user vibe and not notice what needs to improve as much. We need to
> > make the new user experience awesome.
> >
> > Thanks
> > Joe
> > On Nov 2, 2015 10:08 AM, "Mark Petronic" <ma...@gmail.com> wrote:
> >
> > > My primary use is for understanding Nifi. I like to direct various
> > > processors output into both their logical next processor stage as well
> as
> > > into a log attribute processor. Then I tail the Nifi app log file and
> > watch
> > > what happens - in real time. I do not intend to use this for long term
> > log
> > > retention. I agree that providence is the right choice for that. So,
> the
> > > only reason I wanted to allow configuration of a custom logger was
> simply
> > > to isolate all the attribute-rich logging from the normal logging
> > because I
> > > was primarily interested in the attribute flows as a way to (a) better
> > > understand what a processor emits because, frankly, the documentation
> of
> > > some of the processors is very sparse. So, I learn imperatively, so to
> > > speak. I say that as a new user. I feel I should be able to get a
> pretty
> > > good understanding of a processor by reading the usage. But I am
> finding
> > > that the documentation, in some cases, is more like what I like to
> refer
> > to
> > > as, "note to self" documentation. Great if you are the guy who wrote
> the
> > > processor with those "insights" - not so great if you are not the
> > > developer. So, then I need to dig up the code. That should not be
> needed
> > as
> > > the first step of understanding a processor as a new user. There is
> some
> > > well documented processors but not all are, IMHO. (b) Validate my flows
> > > with some test data and verify attribute values look correct and
> routing
> > is
> > > happen on them as expected, etc. Again, easier, IMO, to see in the logs
> > > than digging into the providence data.
> > >
> > > Maybe this is just a good "private" feature for me so maybe I will just
> > > create a private version to use on my own. I already have it working
> but
> > > would need more polish to achieve PR status. Maybe this is the sort of
> > > thing that others would not find beneficial? That's fine. There are
> > others
> > > ways I can contribute in the future. I'm still having fun! :)
> > >
> > > On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <jo...@gmail.com> wrote:
> > >
> > > > Mark Petronic,
> > > >
> > > > I share Payne's perspective on this. But I'd also like to work with
> > > > you to better understand the workflow. For those of us that have
> used
> > > > this tool for a long time there is a lot we take for granted from a
> > > > new user perspective. We believe the provenance feature to provide a
> > > > far superior option to understanding how an item went through the
> > > > system and the timing and what we knew when and so on. But, it would
> > > > be great to understand it from your perspective as someone learning
> > > > NiFi. Not meaning to take away from your proposed contrib - that
> > > > would be great too. Just want to see if the prov user experience
> > > > solves what you're looking for and if not can we make it do that.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <ma...@hotmail.com>
> > > wrote:
> > > > > Mark,
> > > > >
> > > > > To make sure that I understand what you're proposing, you want to
> > add a
> > > > property to
> > > > > LogAttribute that allows users to provide a custom logger name?
> > > > >
> > > > > If that is indeed what you are suggesting then I think it's a great
> > > idea.
> > > > >
> > > > > That being said, in practice I rarely ever use LogAttribute and we
> > even
> > > > considered removing
> > > > > it from the codebase before we open sourced, because the Data
> > > Provenance
> > > > provides a
> > > > > much better view of what's going on to debug your flows.
> > > > >
> > > > > I know you're pretty new to NiFi, so if you've not yet had a chance
> > to
> > > > play with the Provenance,
> > > > > you can see the section in the User Guide at
> > > >
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > > <
> > > >
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > > >
> > > > >
> > > > > If you're interested in updating the LogAttribute processor,
> though,
> > > > we'd be happy to have
> > > > > that contribution added, as it does make the Processor more usable.
> > > > >
> > > > > Thanks
> > > > > -Mark
> > > > >
> > > > >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <
> markpetronic@gmail.com
> > >
> > > > wrote:
> > > > >>
> > > > >> From the code, it appears it cannot be done as the attribute
> logging
> > > > >> goes the same getLogger() instance as the normal nifi-app traces.
> > Has
> > > > >> anyone considered making that configurable, maybe allowing you do
> > > > >> define a different logger name for LogAttribute then creating that
> > > > >> logger definition in log back conf allowing flexibility? I'm using
> > > > >> attribute logging heavily as I try to better learn/debug Nifi (it
> > give
> > > > >> you a nice 'under the hood' view of the flow) and build up some
> > flows
> > > > >> and feel it would be beneficial to be able to capture the
> > LogAttribte
> > > > >> message by themselves for more clarity on what is happening. I
> would
> > > > >> not mind maybe trying to implement this feature as my first crack
> at
> > > > >> contributing to the project. Seems like a fairly easy one that
> would
> > > > >> allow me to "go through the motions" of a full pull request
> process
> > > > >> and iron out the process. Anyone have any thoughts on this?
> > > > >
> > > >
> > >
> >
>
Re: LogAttribute - Sending that output to a custom logger?
Posted by Mark Petronic <ma...@gmail.com>.
I'm sort of in the camp of "don't come with a complaint if you don't come
with a solution" and hesitated to even raise the documentation comment
without just fixing it myself. How about this, I just do some updates on
some processor docs myself and use that as my first contribution to work
through the process of committing to this project?
But, to give you one quick example, EvaluateJSONPath (which, btw has pretty
good docs otherwise) does not mention HOW to extract the JSON you are
interested in. I had to look at the code to figure out it used this:
https://github.com/jayway/JsonPath. Ok, that was not hard, I admit, but, as
a user, should I need to look at the code for such information? I submit,
no. Me personally, I like to dig into the code. So, this is more a comment
on "overall goodness" for the general new user experience.
I agree with your assessment of 'new user vibe' as I am starting to not
notice it as much. lol
On Mon, Nov 2, 2015 at 10:15 AM, Joe Witt <jo...@gmail.com> wrote:
> Mark
>
> All fair points. Can you please point out which processor docs
> specifically should be better. Let's fix em..you will quickly lose that
> new user vibe and not notice what needs to improve as much. We need to
> make the new user experience awesome.
>
> Thanks
> Joe
> On Nov 2, 2015 10:08 AM, "Mark Petronic" <ma...@gmail.com> wrote:
>
> > My primary use is for understanding Nifi. I like to direct various
> > processors output into both their logical next processor stage as well as
> > into a log attribute processor. Then I tail the Nifi app log file and
> watch
> > what happens - in real time. I do not intend to use this for long term
> log
> > retention. I agree that providence is the right choice for that. So, the
> > only reason I wanted to allow configuration of a custom logger was simply
> > to isolate all the attribute-rich logging from the normal logging
> because I
> > was primarily interested in the attribute flows as a way to (a) better
> > understand what a processor emits because, frankly, the documentation of
> > some of the processors is very sparse. So, I learn imperatively, so to
> > speak. I say that as a new user. I feel I should be able to get a pretty
> > good understanding of a processor by reading the usage. But I am finding
> > that the documentation, in some cases, is more like what I like to refer
> to
> > as, "note to self" documentation. Great if you are the guy who wrote the
> > processor with those "insights" - not so great if you are not the
> > developer. So, then I need to dig up the code. That should not be needed
> as
> > the first step of understanding a processor as a new user. There is some
> > well documented processors but not all are, IMHO. (b) Validate my flows
> > with some test data and verify attribute values look correct and routing
> is
> > happen on them as expected, etc. Again, easier, IMO, to see in the logs
> > than digging into the providence data.
> >
> > Maybe this is just a good "private" feature for me so maybe I will just
> > create a private version to use on my own. I already have it working but
> > would need more polish to achieve PR status. Maybe this is the sort of
> > thing that others would not find beneficial? That's fine. There are
> others
> > ways I can contribute in the future. I'm still having fun! :)
> >
> > On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <jo...@gmail.com> wrote:
> >
> > > Mark Petronic,
> > >
> > > I share Payne's perspective on this. But I'd also like to work with
> > > you to better understand the workflow. For those of us that have used
> > > this tool for a long time there is a lot we take for granted from a
> > > new user perspective. We believe the provenance feature to provide a
> > > far superior option to understanding how an item went through the
> > > system and the timing and what we knew when and so on. But, it would
> > > be great to understand it from your perspective as someone learning
> > > NiFi. Not meaning to take away from your proposed contrib - that
> > > would be great too. Just want to see if the prov user experience
> > > solves what you're looking for and if not can we make it do that.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <ma...@hotmail.com>
> > wrote:
> > > > Mark,
> > > >
> > > > To make sure that I understand what you're proposing, you want to
> add a
> > > property to
> > > > LogAttribute that allows users to provide a custom logger name?
> > > >
> > > > If that is indeed what you are suggesting then I think it's a great
> > idea.
> > > >
> > > > That being said, in practice I rarely ever use LogAttribute and we
> even
> > > considered removing
> > > > it from the codebase before we open sourced, because the Data
> > Provenance
> > > provides a
> > > > much better view of what's going on to debug your flows.
> > > >
> > > > I know you're pretty new to NiFi, so if you've not yet had a chance
> to
> > > play with the Provenance,
> > > > you can see the section in the User Guide at
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > <
> > >
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > > >
> > > >
> > > > If you're interested in updating the LogAttribute processor, though,
> > > we'd be happy to have
> > > > that contribution added, as it does make the Processor more usable.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <markpetronic@gmail.com
> >
> > > wrote:
> > > >>
> > > >> From the code, it appears it cannot be done as the attribute logging
> > > >> goes the same getLogger() instance as the normal nifi-app traces.
> Has
> > > >> anyone considered making that configurable, maybe allowing you do
> > > >> define a different logger name for LogAttribute then creating that
> > > >> logger definition in log back conf allowing flexibility? I'm using
> > > >> attribute logging heavily as I try to better learn/debug Nifi (it
> give
> > > >> you a nice 'under the hood' view of the flow) and build up some
> flows
> > > >> and feel it would be beneficial to be able to capture the
> LogAttribte
> > > >> message by themselves for more clarity on what is happening. I would
> > > >> not mind maybe trying to implement this feature as my first crack at
> > > >> contributing to the project. Seems like a fairly easy one that would
> > > >> allow me to "go through the motions" of a full pull request process
> > > >> and iron out the process. Anyone have any thoughts on this?
> > > >
> > >
> >
>
Re: LogAttribute - Sending that output to a custom logger?
Posted by Joe Witt <jo...@gmail.com>.
Mark
All fair points. Can you please point out which processor docs
specifically should be better. Let's fix em..you will quickly lose that
new user vibe and not notice what needs to improve as much. We need to
make the new user experience awesome.
Thanks
Joe
On Nov 2, 2015 10:08 AM, "Mark Petronic" <ma...@gmail.com> wrote:
> My primary use is for understanding Nifi. I like to direct various
> processors output into both their logical next processor stage as well as
> into a log attribute processor. Then I tail the Nifi app log file and watch
> what happens - in real time. I do not intend to use this for long term log
> retention. I agree that providence is the right choice for that. So, the
> only reason I wanted to allow configuration of a custom logger was simply
> to isolate all the attribute-rich logging from the normal logging because I
> was primarily interested in the attribute flows as a way to (a) better
> understand what a processor emits because, frankly, the documentation of
> some of the processors is very sparse. So, I learn imperatively, so to
> speak. I say that as a new user. I feel I should be able to get a pretty
> good understanding of a processor by reading the usage. But I am finding
> that the documentation, in some cases, is more like what I like to refer to
> as, "note to self" documentation. Great if you are the guy who wrote the
> processor with those "insights" - not so great if you are not the
> developer. So, then I need to dig up the code. That should not be needed as
> the first step of understanding a processor as a new user. There is some
> well documented processors but not all are, IMHO. (b) Validate my flows
> with some test data and verify attribute values look correct and routing is
> happen on them as expected, etc. Again, easier, IMO, to see in the logs
> than digging into the providence data.
>
> Maybe this is just a good "private" feature for me so maybe I will just
> create a private version to use on my own. I already have it working but
> would need more polish to achieve PR status. Maybe this is the sort of
> thing that others would not find beneficial? That's fine. There are others
> ways I can contribute in the future. I'm still having fun! :)
>
> On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <jo...@gmail.com> wrote:
>
> > Mark Petronic,
> >
> > I share Payne's perspective on this. But I'd also like to work with
> > you to better understand the workflow. For those of us that have used
> > this tool for a long time there is a lot we take for granted from a
> > new user perspective. We believe the provenance feature to provide a
> > far superior option to understanding how an item went through the
> > system and the timing and what we knew when and so on. But, it would
> > be great to understand it from your perspective as someone learning
> > NiFi. Not meaning to take away from your proposed contrib - that
> > would be great too. Just want to see if the prov user experience
> > solves what you're looking for and if not can we make it do that.
> >
> > Thanks
> > Joe
> >
> > On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <ma...@hotmail.com>
> wrote:
> > > Mark,
> > >
> > > To make sure that I understand what you're proposing, you want to add a
> > property to
> > > LogAttribute that allows users to provide a custom logger name?
> > >
> > > If that is indeed what you are suggesting then I think it's a great
> idea.
> > >
> > > That being said, in practice I rarely ever use LogAttribute and we even
> > considered removing
> > > it from the codebase before we open sourced, because the Data
> Provenance
> > provides a
> > > much better view of what's going on to debug your flows.
> > >
> > > I know you're pretty new to NiFi, so if you've not yet had a chance to
> > play with the Provenance,
> > > you can see the section in the User Guide at
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > <
> >
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> > >
> > >
> > > If you're interested in updating the LogAttribute processor, though,
> > we'd be happy to have
> > > that contribution added, as it does make the Processor more usable.
> > >
> > > Thanks
> > > -Mark
> > >
> > >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <ma...@gmail.com>
> > wrote:
> > >>
> > >> From the code, it appears it cannot be done as the attribute logging
> > >> goes the same getLogger() instance as the normal nifi-app traces. Has
> > >> anyone considered making that configurable, maybe allowing you do
> > >> define a different logger name for LogAttribute then creating that
> > >> logger definition in log back conf allowing flexibility? I'm using
> > >> attribute logging heavily as I try to better learn/debug Nifi (it give
> > >> you a nice 'under the hood' view of the flow) and build up some flows
> > >> and feel it would be beneficial to be able to capture the LogAttribte
> > >> message by themselves for more clarity on what is happening. I would
> > >> not mind maybe trying to implement this feature as my first crack at
> > >> contributing to the project. Seems like a fairly easy one that would
> > >> allow me to "go through the motions" of a full pull request process
> > >> and iron out the process. Anyone have any thoughts on this?
> > >
> >
>
Re: LogAttribute - Sending that output to a custom logger?
Posted by Mark Petronic <ma...@gmail.com>.
My primary use is for understanding Nifi. I like to direct various
processors output into both their logical next processor stage as well as
into a log attribute processor. Then I tail the Nifi app log file and watch
what happens - in real time. I do not intend to use this for long term log
retention. I agree that providence is the right choice for that. So, the
only reason I wanted to allow configuration of a custom logger was simply
to isolate all the attribute-rich logging from the normal logging because I
was primarily interested in the attribute flows as a way to (a) better
understand what a processor emits because, frankly, the documentation of
some of the processors is very sparse. So, I learn imperatively, so to
speak. I say that as a new user. I feel I should be able to get a pretty
good understanding of a processor by reading the usage. But I am finding
that the documentation, in some cases, is more like what I like to refer to
as, "note to self" documentation. Great if you are the guy who wrote the
processor with those "insights" - not so great if you are not the
developer. So, then I need to dig up the code. That should not be needed as
the first step of understanding a processor as a new user. There is some
well documented processors but not all are, IMHO. (b) Validate my flows
with some test data and verify attribute values look correct and routing is
happen on them as expected, etc. Again, easier, IMO, to see in the logs
than digging into the providence data.
Maybe this is just a good "private" feature for me so maybe I will just
create a private version to use on my own. I already have it working but
would need more polish to achieve PR status. Maybe this is the sort of
thing that others would not find beneficial? That's fine. There are others
ways I can contribute in the future. I'm still having fun! :)
On Sun, Nov 1, 2015 at 12:41 PM, Joe Witt <jo...@gmail.com> wrote:
> Mark Petronic,
>
> I share Payne's perspective on this. But I'd also like to work with
> you to better understand the workflow. For those of us that have used
> this tool for a long time there is a lot we take for granted from a
> new user perspective. We believe the provenance feature to provide a
> far superior option to understanding how an item went through the
> system and the timing and what we knew when and so on. But, it would
> be great to understand it from your perspective as someone learning
> NiFi. Not meaning to take away from your proposed contrib - that
> would be great too. Just want to see if the prov user experience
> solves what you're looking for and if not can we make it do that.
>
> Thanks
> Joe
>
> On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <ma...@hotmail.com> wrote:
> > Mark,
> >
> > To make sure that I understand what you're proposing, you want to add a
> property to
> > LogAttribute that allows users to provide a custom logger name?
> >
> > If that is indeed what you are suggesting then I think it's a great idea.
> >
> > That being said, in practice I rarely ever use LogAttribute and we even
> considered removing
> > it from the codebase before we open sourced, because the Data Provenance
> provides a
> > much better view of what's going on to debug your flows.
> >
> > I know you're pretty new to NiFi, so if you've not yet had a chance to
> play with the Provenance,
> > you can see the section in the User Guide at
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> <
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance
> >
> >
> > If you're interested in updating the LogAttribute processor, though,
> we'd be happy to have
> > that contribution added, as it does make the Processor more usable.
> >
> > Thanks
> > -Mark
> >
> >> On Oct 31, 2015, at 12:35 PM, Mark Petronic <ma...@gmail.com>
> wrote:
> >>
> >> From the code, it appears it cannot be done as the attribute logging
> >> goes the same getLogger() instance as the normal nifi-app traces. Has
> >> anyone considered making that configurable, maybe allowing you do
> >> define a different logger name for LogAttribute then creating that
> >> logger definition in log back conf allowing flexibility? I'm using
> >> attribute logging heavily as I try to better learn/debug Nifi (it give
> >> you a nice 'under the hood' view of the flow) and build up some flows
> >> and feel it would be beneficial to be able to capture the LogAttribte
> >> message by themselves for more clarity on what is happening. I would
> >> not mind maybe trying to implement this feature as my first crack at
> >> contributing to the project. Seems like a fairly easy one that would
> >> allow me to "go through the motions" of a full pull request process
> >> and iron out the process. Anyone have any thoughts on this?
> >
>
Re: LogAttribute - Sending that output to a custom logger?
Posted by Joe Witt <jo...@gmail.com>.
Mark Petronic,
I share Payne's perspective on this. But I'd also like to work with
you to better understand the workflow. For those of us that have used
this tool for a long time there is a lot we take for granted from a
new user perspective. We believe the provenance feature to provide a
far superior option to understanding how an item went through the
system and the timing and what we knew when and so on. But, it would
be great to understand it from your perspective as someone learning
NiFi. Not meaning to take away from your proposed contrib - that
would be great too. Just want to see if the prov user experience
solves what you're looking for and if not can we make it do that.
Thanks
Joe
On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne <ma...@hotmail.com> wrote:
> Mark,
>
> To make sure that I understand what you're proposing, you want to add a property to
> LogAttribute that allows users to provide a custom logger name?
>
> If that is indeed what you are suggesting then I think it's a great idea.
>
> That being said, in practice I rarely ever use LogAttribute and we even considered removing
> it from the codebase before we open sourced, because the Data Provenance provides a
> much better view of what's going on to debug your flows.
>
> I know you're pretty new to NiFi, so if you've not yet had a chance to play with the Provenance,
> you can see the section in the User Guide at http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance <http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance>
>
> If you're interested in updating the LogAttribute processor, though, we'd be happy to have
> that contribution added, as it does make the Processor more usable.
>
> Thanks
> -Mark
>
>> On Oct 31, 2015, at 12:35 PM, Mark Petronic <ma...@gmail.com> wrote:
>>
>> From the code, it appears it cannot be done as the attribute logging
>> goes the same getLogger() instance as the normal nifi-app traces. Has
>> anyone considered making that configurable, maybe allowing you do
>> define a different logger name for LogAttribute then creating that
>> logger definition in log back conf allowing flexibility? I'm using
>> attribute logging heavily as I try to better learn/debug Nifi (it give
>> you a nice 'under the hood' view of the flow) and build up some flows
>> and feel it would be beneficial to be able to capture the LogAttribte
>> message by themselves for more clarity on what is happening. I would
>> not mind maybe trying to implement this feature as my first crack at
>> contributing to the project. Seems like a fairly easy one that would
>> allow me to "go through the motions" of a full pull request process
>> and iron out the process. Anyone have any thoughts on this?
>
Re: LogAttribute - Sending that output to a custom logger?
Posted by Mark Payne <ma...@hotmail.com>.
Mark,
To make sure that I understand what you're proposing, you want to add a property to
LogAttribute that allows users to provide a custom logger name?
If that is indeed what you are suggesting then I think it's a great idea.
That being said, in practice I rarely ever use LogAttribute and we even considered removing
it from the codebase before we open sourced, because the Data Provenance provides a
much better view of what's going on to debug your flows.
I know you're pretty new to NiFi, so if you've not yet had a chance to play with the Provenance,
you can see the section in the User Guide at http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance <http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance>
If you're interested in updating the LogAttribute processor, though, we'd be happy to have
that contribution added, as it does make the Processor more usable.
Thanks
-Mark
> On Oct 31, 2015, at 12:35 PM, Mark Petronic <ma...@gmail.com> wrote:
>
> From the code, it appears it cannot be done as the attribute logging
> goes the same getLogger() instance as the normal nifi-app traces. Has
> anyone considered making that configurable, maybe allowing you do
> define a different logger name for LogAttribute then creating that
> logger definition in log back conf allowing flexibility? I'm using
> attribute logging heavily as I try to better learn/debug Nifi (it give
> you a nice 'under the hood' view of the flow) and build up some flows
> and feel it would be beneficial to be able to capture the LogAttribte
> message by themselves for more clarity on what is happening. I would
> not mind maybe trying to implement this feature as my first crack at
> contributing to the project. Seems like a fairly easy one that would
> allow me to "go through the motions" of a full pull request process
> and iron out the process. Anyone have any thoughts on this?