You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "dam6923 ." <da...@gmail.com> on 2017/12/08 22:38:52 UTC
Moving To SLF4J and Log4J2
Hello Team,
Maybe it's time to finally realize the dream of HBASE-10092
Once the logging infrastructure is upgraded, I would be happy to
assist in reviewing and improving the log messages. I've done a bit
of similar work for the Hive project. It's tedious but doesn't
require deep knowledge of the overall project and can be done
piecemeal.
Commons Logging states that it requires one to use code guards for
logging debug/trace statements. However, SLF4J recommends using
parameters for logging which is faster than guards, and I personally
think it's a cleaner approach. The logging parameters are only
resolved if the log level is appropriate and therefore guards are
often unnecessary. It removes code (the guards) and complexity from
the application code.
This seems like a straight-forward path for the project for eking out
some performance.
https://www.slf4j.org/faq.html#logging_performance
https://commons.apache.org/proper/commons-logging/guide.html#Code_Guards
Thanks.
Re: Moving To SLF4J and Log4J2
Posted by Mike Drob <md...@apache.org>.
I remember doing the research for this many moons ago on a different
project, and dynamically setting log levels (like we do via web ui) is
simply not supported in slf4j.
On Tue, Dec 19, 2017 at 9:29 AM, Balazs Meszaros <
balazs.meszaros@cloudera.com> wrote:
> Thanks for reviewing Appy!
>
> 1. I tried to verify it, log level changes take place through the web ui.
> 2. I put back fatals.
> 3. The property files are still compatible, because I have not updated
> log4j to log4j2 yet. But they won't be compatible after the update.
> 4. I also updated those projects.
>
> Unfortunately there are some issues which need to be solved before updating
> to log4j2:
> 1. There are still some references in our java files to log4j (e.g.
> LogManager references). We use it to set log levels from the code:
> - on the web ui,
> - in the unit tests (I removed them, because there were no asserts on the
> log messages),
> - some command line tools also configured log levels from the code.
> 2. hbase-http also uses log4j for request logging.
>
> If we can't rid off from these dependencies, then our codebase won't be
> completely independent from the logging implementation.
>
> Best regards,
> Balazs
>
> On Tue, Dec 19, 2017 at 2:37 AM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
>
> > Thanks for the ping here Stack. Posted review on the jira.
> > Summary is, we need at least:
> > 1) basic verification
> > 2) fatal markers
> > 3) clear picture on properties file: Is old one compatible? if not, we
> need
> > new ones and document what's breaking. New ones and documentation can be
> > done in followup, but we need clear picture now.
> > 4) Followup jira for hbase-backup/http or a reason why they can't be
> done.
> >
> > Reverting this patch would be hell, wouldn't want to do it because we
> > missed basic checks.
> >
> > -- Appy
> >
> >
> > On Mon, Dec 18, 2017 at 3:53 PM, Apekshit Sharma <ap...@cloudera.com>
> > wrote:
> >
> > > Oh, just 60 pages of review :D
> > >
> > > -- Appy
> > >
> > > On Mon, Dec 18, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Wed, Dec 13, 2017 at 10:09 AM, Stack <st...@duboce.net> wrote:
> > >>
> > >> > On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma <
> appy@cloudera.com>
> > >> > wrote:
> > >> >
> > >> >> Seems like good idea:
> > >> >> - remove long dead dependency
> > >> >> - a bit cleaner code
> > >> >> - hadoop also moved to slf4j
> > >> >>
> > >> >> Quickly looking at codebase to get idea of amount of work required,
> > >> here
> > >> >> are some numbers:
> > >> >> - LOG.debug : ~1800
> > >> >> - LOG.trace : ~500
> > >> >> - LOG.info: ~3000
> > >> >>
> > >> >> Looking at this patch (
> > >> >> https://issues.apache.org/jira/secure/attachment/12901002/
> > >> >> HBASE-19449.1.patch),
> > >> >> seemed like tedious and repetitive task, was wondering if someone
> has
> > >> >> automated it already.
> > >> >> Looks like this can help reduce a huge part of grunt work:
> > >> >> https://www.slf4j.org/migrator.html.
> > >> >>
> > >> >> But before progressing, as a basic validation, can we see:
> > >> >> - an example of old vs new log lines (that there is no diff, or we
> > are
> > >> >> comfortable with what's there)
> > >> >> - an example of changes in properties file
> > >> >>
> > >> >> Maybe starting with hbase-examples module for quick POC.
> > >> >>
> > >> >>
> > >> > I like your suggestion Appy,
> > >> > S
> > >> >
> > >> >
> > >> Balazs has a patch up on HBASE-10092 making the move. Intend to commit
> > it
> > >> in next day or so unless objection.
> > >> S
> > >>
> > >>
> > >>
> > >> >
> > >> >
> > >> >> -- Appy
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > -- Appy
> > >
> >
> >
> >
> > --
> >
> > -- Appy
> >
>
Re: Moving To SLF4J and Log4J2
Posted by Stack <st...@duboce.net>.
sfl4j as our front-end logger just went in (HBASE-10092) thanks to Balazs
Meszaros code-fu and our Appy review (with some timely input from our
comrades over in slf4j2).
There'll be some teething issues to be addressed in the next few days but
going forward, from here on out, as we go, we need to convert as we go our
log messages from being 'guarded' -- i.e. if (LOG.isDebugEnabled...) -- to
instead being parameterized log messages. e.g. the latter rather than the
former in the below:
logger.debug("The new entry is "+entry+".");
logger.debug("The new entry is {}.", entry);
See [1] for background on perf benefits.
Note, FATAL log level is not present in slf4j.
Next up, changing our logging backend (probably log4j2 after we figure how
to bring our operators over from a .properties to an .xml config file --
hbase3?).
Yours,
S
1. https://www.slf4j.org/faq.html#logging_performance
On Tue, Dec 19, 2017 at 9:19 AM, Stack <st...@duboce.net> wrote:
> On Tue, Dec 19, 2017 at 7:29 AM, Balazs Meszaros <
> balazs.meszaros@cloudera.com> wrote:
>
>> Thanks for reviewing Appy!
>>
>> 1. I tried to verify it, log level changes take place through the web ui.
>> 2. I put back fatals.
>> 3. The property files are still compatible, because I have not updated
>> log4j to log4j2 yet. But they won't be compatible after the update.
>>
>
> We are not going to log4j2. You thinking otherwise Balazs?
>
>
>
>> 4. I also updated those projects.
>>
>> Unfortunately there are some issues which need to be solved before
>> updating
>> to log4j2:
>> 1. There are still some references in our java files to log4j (e.g.
>> LogManager references). We use it to set log levels from the code:
>> - on the web ui,
>>
>
> Looking at hadoop3, they seem to keep log4j around in the loglevel servlet
> at least just for changing log levels. We can do same?
>
>
>> - in the unit tests (I removed them, because there were no asserts on
>> the
>> log messages),
>> - some command line tools also configured log levels from the code.
>> 2. hbase-http also uses log4j for request logging.
>>
>> There are downsides to having two systems? hbase-http at least is in its
> module.
>
>
>> If we can't rid off from these dependencies, then our codebase won't be
>> completely independent from the logging implementation.
>>
>>
> Sounds like we can't do a pure cut-over, more like a 95%. Thats good
> enough i'd say.
>
> Thanks for the work Balazs,
> St.Ack
>
>
>
>
>> Best regards,
>> Balazs
>>
>> On Tue, Dec 19, 2017 at 2:37 AM, Apekshit Sharma <ap...@cloudera.com>
>> wrote:
>>
>> > Thanks for the ping here Stack. Posted review on the jira.
>> > Summary is, we need at least:
>> > 1) basic verification
>> > 2) fatal markers
>> > 3) clear picture on properties file: Is old one compatible? if not, we
>> need
>> > new ones and document what's breaking. New ones and documentation can be
>> > done in followup, but we need clear picture now.
>> > 4) Followup jira for hbase-backup/http or a reason why they can't be
>> done.
>> >
>> > Reverting this patch would be hell, wouldn't want to do it because we
>> > missed basic checks.
>> >
>> > -- Appy
>> >
>> >
>> > On Mon, Dec 18, 2017 at 3:53 PM, Apekshit Sharma <ap...@cloudera.com>
>> > wrote:
>> >
>> > > Oh, just 60 pages of review :D
>> > >
>> > > -- Appy
>> > >
>> > > On Mon, Dec 18, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
>> > >
>> > >> On Wed, Dec 13, 2017 at 10:09 AM, Stack <st...@duboce.net> wrote:
>> > >>
>> > >> > On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma <
>> appy@cloudera.com>
>> > >> > wrote:
>> > >> >
>> > >> >> Seems like good idea:
>> > >> >> - remove long dead dependency
>> > >> >> - a bit cleaner code
>> > >> >> - hadoop also moved to slf4j
>> > >> >>
>> > >> >> Quickly looking at codebase to get idea of amount of work
>> required,
>> > >> here
>> > >> >> are some numbers:
>> > >> >> - LOG.debug : ~1800
>> > >> >> - LOG.trace : ~500
>> > >> >> - LOG.info: ~3000
>> > >> >>
>> > >> >> Looking at this patch (
>> > >> >> https://issues.apache.org/jira/secure/attachment/12901002/
>> > >> >> HBASE-19449.1.patch),
>> > >> >> seemed like tedious and repetitive task, was wondering if someone
>> has
>> > >> >> automated it already.
>> > >> >> Looks like this can help reduce a huge part of grunt work:
>> > >> >> https://www.slf4j.org/migrator.html.
>> > >> >>
>> > >> >> But before progressing, as a basic validation, can we see:
>> > >> >> - an example of old vs new log lines (that there is no diff, or we
>> > are
>> > >> >> comfortable with what's there)
>> > >> >> - an example of changes in properties file
>> > >> >>
>> > >> >> Maybe starting with hbase-examples module for quick POC.
>> > >> >>
>> > >> >>
>> > >> > I like your suggestion Appy,
>> > >> > S
>> > >> >
>> > >> >
>> > >> Balazs has a patch up on HBASE-10092 making the move. Intend to
>> commit
>> > it
>> > >> in next day or so unless objection.
>> > >> S
>> > >>
>> > >>
>> > >>
>> > >> >
>> > >> >
>> > >> >> -- Appy
>> > >> >>
>> > >> >
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > -- Appy
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > -- Appy
>> >
>>
>
>
Re: Moving To SLF4J and Log4J2
Posted by Stack <st...@duboce.net>.
On Tue, Dec 19, 2017 at 7:29 AM, Balazs Meszaros <
balazs.meszaros@cloudera.com> wrote:
> Thanks for reviewing Appy!
>
> 1. I tried to verify it, log level changes take place through the web ui.
> 2. I put back fatals.
> 3. The property files are still compatible, because I have not updated
> log4j to log4j2 yet. But they won't be compatible after the update.
>
We are not going to log4j2. You thinking otherwise Balazs?
> 4. I also updated those projects.
>
> Unfortunately there are some issues which need to be solved before updating
> to log4j2:
> 1. There are still some references in our java files to log4j (e.g.
> LogManager references). We use it to set log levels from the code:
> - on the web ui,
>
Looking at hadoop3, they seem to keep log4j around in the loglevel servlet
at least just for changing log levels. We can do same?
> - in the unit tests (I removed them, because there were no asserts on the
> log messages),
> - some command line tools also configured log levels from the code.
> 2. hbase-http also uses log4j for request logging.
>
> There are downsides to having two systems? hbase-http at least is in its
module.
> If we can't rid off from these dependencies, then our codebase won't be
> completely independent from the logging implementation.
>
>
Sounds like we can't do a pure cut-over, more like a 95%. Thats good enough
i'd say.
Thanks for the work Balazs,
St.Ack
> Best regards,
> Balazs
>
> On Tue, Dec 19, 2017 at 2:37 AM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
>
> > Thanks for the ping here Stack. Posted review on the jira.
> > Summary is, we need at least:
> > 1) basic verification
> > 2) fatal markers
> > 3) clear picture on properties file: Is old one compatible? if not, we
> need
> > new ones and document what's breaking. New ones and documentation can be
> > done in followup, but we need clear picture now.
> > 4) Followup jira for hbase-backup/http or a reason why they can't be
> done.
> >
> > Reverting this patch would be hell, wouldn't want to do it because we
> > missed basic checks.
> >
> > -- Appy
> >
> >
> > On Mon, Dec 18, 2017 at 3:53 PM, Apekshit Sharma <ap...@cloudera.com>
> > wrote:
> >
> > > Oh, just 60 pages of review :D
> > >
> > > -- Appy
> > >
> > > On Mon, Dec 18, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Wed, Dec 13, 2017 at 10:09 AM, Stack <st...@duboce.net> wrote:
> > >>
> > >> > On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma <
> appy@cloudera.com>
> > >> > wrote:
> > >> >
> > >> >> Seems like good idea:
> > >> >> - remove long dead dependency
> > >> >> - a bit cleaner code
> > >> >> - hadoop also moved to slf4j
> > >> >>
> > >> >> Quickly looking at codebase to get idea of amount of work required,
> > >> here
> > >> >> are some numbers:
> > >> >> - LOG.debug : ~1800
> > >> >> - LOG.trace : ~500
> > >> >> - LOG.info: ~3000
> > >> >>
> > >> >> Looking at this patch (
> > >> >> https://issues.apache.org/jira/secure/attachment/12901002/
> > >> >> HBASE-19449.1.patch),
> > >> >> seemed like tedious and repetitive task, was wondering if someone
> has
> > >> >> automated it already.
> > >> >> Looks like this can help reduce a huge part of grunt work:
> > >> >> https://www.slf4j.org/migrator.html.
> > >> >>
> > >> >> But before progressing, as a basic validation, can we see:
> > >> >> - an example of old vs new log lines (that there is no diff, or we
> > are
> > >> >> comfortable with what's there)
> > >> >> - an example of changes in properties file
> > >> >>
> > >> >> Maybe starting with hbase-examples module for quick POC.
> > >> >>
> > >> >>
> > >> > I like your suggestion Appy,
> > >> > S
> > >> >
> > >> >
> > >> Balazs has a patch up on HBASE-10092 making the move. Intend to commit
> > it
> > >> in next day or so unless objection.
> > >> S
> > >>
> > >>
> > >>
> > >> >
> > >> >
> > >> >> -- Appy
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > -- Appy
> > >
> >
> >
> >
> > --
> >
> > -- Appy
> >
>
Re: Moving To SLF4J and Log4J2
Posted by Balazs Meszaros <ba...@cloudera.com>.
Thanks for reviewing Appy!
1. I tried to verify it, log level changes take place through the web ui.
2. I put back fatals.
3. The property files are still compatible, because I have not updated
log4j to log4j2 yet. But they won't be compatible after the update.
4. I also updated those projects.
Unfortunately there are some issues which need to be solved before updating
to log4j2:
1. There are still some references in our java files to log4j (e.g.
LogManager references). We use it to set log levels from the code:
- on the web ui,
- in the unit tests (I removed them, because there were no asserts on the
log messages),
- some command line tools also configured log levels from the code.
2. hbase-http also uses log4j for request logging.
If we can't rid off from these dependencies, then our codebase won't be
completely independent from the logging implementation.
Best regards,
Balazs
On Tue, Dec 19, 2017 at 2:37 AM, Apekshit Sharma <ap...@cloudera.com> wrote:
> Thanks for the ping here Stack. Posted review on the jira.
> Summary is, we need at least:
> 1) basic verification
> 2) fatal markers
> 3) clear picture on properties file: Is old one compatible? if not, we need
> new ones and document what's breaking. New ones and documentation can be
> done in followup, but we need clear picture now.
> 4) Followup jira for hbase-backup/http or a reason why they can't be done.
>
> Reverting this patch would be hell, wouldn't want to do it because we
> missed basic checks.
>
> -- Appy
>
>
> On Mon, Dec 18, 2017 at 3:53 PM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
>
> > Oh, just 60 pages of review :D
> >
> > -- Appy
> >
> > On Mon, Dec 18, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Wed, Dec 13, 2017 at 10:09 AM, Stack <st...@duboce.net> wrote:
> >>
> >> > On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma <ap...@cloudera.com>
> >> > wrote:
> >> >
> >> >> Seems like good idea:
> >> >> - remove long dead dependency
> >> >> - a bit cleaner code
> >> >> - hadoop also moved to slf4j
> >> >>
> >> >> Quickly looking at codebase to get idea of amount of work required,
> >> here
> >> >> are some numbers:
> >> >> - LOG.debug : ~1800
> >> >> - LOG.trace : ~500
> >> >> - LOG.info: ~3000
> >> >>
> >> >> Looking at this patch (
> >> >> https://issues.apache.org/jira/secure/attachment/12901002/
> >> >> HBASE-19449.1.patch),
> >> >> seemed like tedious and repetitive task, was wondering if someone has
> >> >> automated it already.
> >> >> Looks like this can help reduce a huge part of grunt work:
> >> >> https://www.slf4j.org/migrator.html.
> >> >>
> >> >> But before progressing, as a basic validation, can we see:
> >> >> - an example of old vs new log lines (that there is no diff, or we
> are
> >> >> comfortable with what's there)
> >> >> - an example of changes in properties file
> >> >>
> >> >> Maybe starting with hbase-examples module for quick POC.
> >> >>
> >> >>
> >> > I like your suggestion Appy,
> >> > S
> >> >
> >> >
> >> Balazs has a patch up on HBASE-10092 making the move. Intend to commit
> it
> >> in next day or so unless objection.
> >> S
> >>
> >>
> >>
> >> >
> >> >
> >> >> -- Appy
> >> >>
> >> >
> >> >
> >>
> >
> >
> >
> > --
> >
> > -- Appy
> >
>
>
>
> --
>
> -- Appy
>
Re: Moving To SLF4J and Log4J2
Posted by Apekshit Sharma <ap...@cloudera.com>.
Thanks for the ping here Stack. Posted review on the jira.
Summary is, we need at least:
1) basic verification
2) fatal markers
3) clear picture on properties file: Is old one compatible? if not, we need
new ones and document what's breaking. New ones and documentation can be
done in followup, but we need clear picture now.
4) Followup jira for hbase-backup/http or a reason why they can't be done.
Reverting this patch would be hell, wouldn't want to do it because we
missed basic checks.
-- Appy
On Mon, Dec 18, 2017 at 3:53 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> Oh, just 60 pages of review :D
>
> -- Appy
>
> On Mon, Dec 18, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
>
>> On Wed, Dec 13, 2017 at 10:09 AM, Stack <st...@duboce.net> wrote:
>>
>> > On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma <ap...@cloudera.com>
>> > wrote:
>> >
>> >> Seems like good idea:
>> >> - remove long dead dependency
>> >> - a bit cleaner code
>> >> - hadoop also moved to slf4j
>> >>
>> >> Quickly looking at codebase to get idea of amount of work required,
>> here
>> >> are some numbers:
>> >> - LOG.debug : ~1800
>> >> - LOG.trace : ~500
>> >> - LOG.info: ~3000
>> >>
>> >> Looking at this patch (
>> >> https://issues.apache.org/jira/secure/attachment/12901002/
>> >> HBASE-19449.1.patch),
>> >> seemed like tedious and repetitive task, was wondering if someone has
>> >> automated it already.
>> >> Looks like this can help reduce a huge part of grunt work:
>> >> https://www.slf4j.org/migrator.html.
>> >>
>> >> But before progressing, as a basic validation, can we see:
>> >> - an example of old vs new log lines (that there is no diff, or we are
>> >> comfortable with what's there)
>> >> - an example of changes in properties file
>> >>
>> >> Maybe starting with hbase-examples module for quick POC.
>> >>
>> >>
>> > I like your suggestion Appy,
>> > S
>> >
>> >
>> Balazs has a patch up on HBASE-10092 making the move. Intend to commit it
>> in next day or so unless objection.
>> S
>>
>>
>>
>> >
>> >
>> >> -- Appy
>> >>
>> >
>> >
>>
>
>
>
> --
>
> -- Appy
>
--
-- Appy
Re: Moving To SLF4J and Log4J2
Posted by Apekshit Sharma <ap...@cloudera.com>.
Oh, just 60 pages of review :D
-- Appy
On Mon, Dec 18, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
> On Wed, Dec 13, 2017 at 10:09 AM, Stack <st...@duboce.net> wrote:
>
> > On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma <ap...@cloudera.com>
> > wrote:
> >
> >> Seems like good idea:
> >> - remove long dead dependency
> >> - a bit cleaner code
> >> - hadoop also moved to slf4j
> >>
> >> Quickly looking at codebase to get idea of amount of work required, here
> >> are some numbers:
> >> - LOG.debug : ~1800
> >> - LOG.trace : ~500
> >> - LOG.info: ~3000
> >>
> >> Looking at this patch (
> >> https://issues.apache.org/jira/secure/attachment/12901002/
> >> HBASE-19449.1.patch),
> >> seemed like tedious and repetitive task, was wondering if someone has
> >> automated it already.
> >> Looks like this can help reduce a huge part of grunt work:
> >> https://www.slf4j.org/migrator.html.
> >>
> >> But before progressing, as a basic validation, can we see:
> >> - an example of old vs new log lines (that there is no diff, or we are
> >> comfortable with what's there)
> >> - an example of changes in properties file
> >>
> >> Maybe starting with hbase-examples module for quick POC.
> >>
> >>
> > I like your suggestion Appy,
> > S
> >
> >
> Balazs has a patch up on HBASE-10092 making the move. Intend to commit it
> in next day or so unless objection.
> S
>
>
>
> >
> >
> >> -- Appy
> >>
> >
> >
>
--
-- Appy
Re: Moving To SLF4J and Log4J2
Posted by Stack <st...@duboce.net>.
On Wed, Dec 13, 2017 at 10:09 AM, Stack <st...@duboce.net> wrote:
> On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma <ap...@cloudera.com>
> wrote:
>
>> Seems like good idea:
>> - remove long dead dependency
>> - a bit cleaner code
>> - hadoop also moved to slf4j
>>
>> Quickly looking at codebase to get idea of amount of work required, here
>> are some numbers:
>> - LOG.debug : ~1800
>> - LOG.trace : ~500
>> - LOG.info: ~3000
>>
>> Looking at this patch (
>> https://issues.apache.org/jira/secure/attachment/12901002/
>> HBASE-19449.1.patch),
>> seemed like tedious and repetitive task, was wondering if someone has
>> automated it already.
>> Looks like this can help reduce a huge part of grunt work:
>> https://www.slf4j.org/migrator.html.
>>
>> But before progressing, as a basic validation, can we see:
>> - an example of old vs new log lines (that there is no diff, or we are
>> comfortable with what's there)
>> - an example of changes in properties file
>>
>> Maybe starting with hbase-examples module for quick POC.
>>
>>
> I like your suggestion Appy,
> S
>
>
Balazs has a patch up on HBASE-10092 making the move. Intend to commit it
in next day or so unless objection.
S
>
>
>> -- Appy
>>
>
>
Re: Moving To SLF4J and Log4J2
Posted by Stack <st...@duboce.net>.
On Mon, Dec 11, 2017 at 12:49 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> Seems like good idea:
> - remove long dead dependency
> - a bit cleaner code
> - hadoop also moved to slf4j
>
> Quickly looking at codebase to get idea of amount of work required, here
> are some numbers:
> - LOG.debug : ~1800
> - LOG.trace : ~500
> - LOG.info: ~3000
>
> Looking at this patch (
> https://issues.apache.org/jira/secure/attachment/
> 12901002/HBASE-19449.1.patch),
> seemed like tedious and repetitive task, was wondering if someone has
> automated it already.
> Looks like this can help reduce a huge part of grunt work:
> https://www.slf4j.org/migrator.html.
>
> But before progressing, as a basic validation, can we see:
> - an example of old vs new log lines (that there is no diff, or we are
> comfortable with what's there)
> - an example of changes in properties file
>
> Maybe starting with hbase-examples module for quick POC.
>
>
I like your suggestion Appy,
S
> -- Appy
>
Re: Moving To SLF4J and Log4J2
Posted by Apekshit Sharma <ap...@cloudera.com>.
Seems like good idea:
- remove long dead dependency
- a bit cleaner code
- hadoop also moved to slf4j
Quickly looking at codebase to get idea of amount of work required, here
are some numbers:
- LOG.debug : ~1800
- LOG.trace : ~500
- LOG.info: ~3000
Looking at this patch (
https://issues.apache.org/jira/secure/attachment/12901002/HBASE-19449.1.patch),
seemed like tedious and repetitive task, was wondering if someone has
automated it already.
Looks like this can help reduce a huge part of grunt work:
https://www.slf4j.org/migrator.html.
But before progressing, as a basic validation, can we see:
- an example of old vs new log lines (that there is no diff, or we are
comfortable with what's there)
- an example of changes in properties file
Maybe starting with hbase-examples module for quick POC.
-- Appy
Re: Moving To SLF4J and Log4J2
Posted by "dam6923 ." <da...@gmail.com>.
Just to clarify, I did not help with the migration... I've been
helping, piecemeal, to review comments for spelling, grammar,
contractions and to remove code guards in favor of parameters.
On Sat, Dec 9, 2017 at 11:22 PM, Stack <st...@duboce.net> wrote:
> On Sat, Dec 9, 2017 at 6:03 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
>
>> +1 for dropping dependency which has been EOL for long now.
>>
>> What does the work here looks like? Change dependency, update properties
>> file, changed log messages, what else?
>>
>>
> The Hadoop issue has some prescription, regexes to run, etc. We should
> gauge it.
> S
>
>
>> Given upcoming beta1 release, what's the minimum work required to change
>> just the dependency? Is it possible to make code change (actual log lines)
>> separately/incrementally?
>>
>> @Beluga: Any gotchas from experiences in Hive?
>>
>> Thanks
>> -- Appy
>>
Re: Moving To SLF4J and Log4J2
Posted by Stack <st...@duboce.net>.
On Sat, Dec 9, 2017 at 6:03 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> +1 for dropping dependency which has been EOL for long now.
>
> What does the work here looks like? Change dependency, update properties
> file, changed log messages, what else?
>
>
The Hadoop issue has some prescription, regexes to run, etc. We should
gauge it.
S
> Given upcoming beta1 release, what's the minimum work required to change
> just the dependency? Is it possible to make code change (actual log lines)
> separately/incrementally?
>
> @Beluga: Any gotchas from experiences in Hive?
>
> Thanks
> -- Appy
>
Re: Moving To SLF4J and Log4J2
Posted by Apekshit Sharma <ap...@cloudera.com>.
+1 for dropping dependency which has been EOL for long now.
What does the work here looks like? Change dependency, update properties
file, changed log messages, what else?
Given upcoming beta1 release, what's the minimum work required to change
just the dependency? Is it possible to make code change (actual log lines)
separately/incrementally?
@Beluga: Any gotchas from experiences in Hive?
Thanks
-- Appy
Re: Moving To SLF4J and Log4J2
Posted by Stack <st...@duboce.net>.
Related, should we move to slf4j for hbase2?
This comes of the issue opened by Beluga:"HBASE-19449 Implement SLF4J and
SLF4J Parameter Substitution"
In the issue, Duo reminded us of the recent hadoop move to slf4j.The issue
is HADOOP-12956. It begins:
"5 August 2015 --The Apache Logging Services™ Project Management Committee
(PMC) has announced that the Log4j™ 1.x logging framework has reached its
end of life (EOL) and is no longer officially supported.
https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
A whole framework log4j2 upgrade has to be synchronized, partly for
improved performance brought about by log4j2.
https://logging.apache.org/log4j/2.x/manual/async.html#Performance"
Other gems in the issue include: "The only thing that supports the log4j 1
properties files is Log4j 1.x. That was declared EOL 2 years ago. The last
release of Log4j 1 was 5 1/2 years ago. It doesn't run in Java 9 without
hacking it."
There is back and forth. log4j2 is a breaking change. It does not support
old-style log4j1 properties files. Slf4j does a better job here. Our Sean
notes:
"SLF4j is exactly the operational answer Hadoop needs. It lets us move our
code's assumptions off of log4j1 while providing a log4j 1 bridge that will
work with existing log4j 1 properties files. That way we can work
incrementally on updating the code base while not requiring operators to
change anything. Once we're done, operators who want to switch early can do
so. As a project we can wait for our next major version to move the default
to some other logging implementation."
The LOG4J2-63 issue for supporting log4j1 properties files has not made
progress.
Hadoop 3 moved to slf4j (and 2.9?).
Should we? (Then Beluga can go to work on his convertions... smile).
St.Ack
On Fri, Dec 8, 2017 at 2:38 PM, dam6923 . <da...@gmail.com> wrote:
> Hello Team,
>
> Maybe it's time to finally realize the dream of HBASE-10092
>
> Once the logging infrastructure is upgraded, I would be happy to
> assist in reviewing and improving the log messages. I've done a bit
> of similar work for the Hive project. It's tedious but doesn't
> require deep knowledge of the overall project and can be done
> piecemeal.
>
> Commons Logging states that it requires one to use code guards for
> logging debug/trace statements. However, SLF4J recommends using
> parameters for logging which is faster than guards, and I personally
> think it's a cleaner approach. The logging parameters are only
> resolved if the log level is appropriate and therefore guards are
> often unnecessary. It removes code (the guards) and complexity from
> the application code.
>
> This seems like a straight-forward path for the project for eking out
> some performance.
>
> https://www.slf4j.org/faq.html#logging_performance
> https://commons.apache.org/proper/commons-logging/guide.html#Code_Guards
>
> Thanks.
>