You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by "张铎(Duo Zhang)" <pa...@gmail.com> on 2022/01/09 15:01:51 UTC
[DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
I brought this up in the incubator mailing list, and was suggested to
report directly to the log4j community.
https://issues.apache.org/jira/browse/HADOOP-16206
In hadoop we started to try migrating to log4j2 long ago, but it is not
easy. For now, one of the most blocker issues is the lack of support of
'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to stay
on the bridge api, we want to fully migrate to log4j2 finally, so
supporting the above grammar in log4j1 bridge is not enough.
This is the main log4j configuration file for hadoop
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
We have this in the file
# Define the root logger to the system property "hadoop.root.logger".
> log4j.rootLogger=${hadoop.root.logger}
So our users could simply pass -Dhadoop.root.logger to control the level
and appender.
https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
Here we use an environment variable in our start up scripts. And I believe
there are lots of other hadoop deployment systems which will have their own
start scripts.
So in general, it is just impossible for us to drop the
-Dhadoop.root.logger way of configuring our logging system.
So here I want to ask if it is possible for log4j2 to still support the
'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
back with no change, we just need a way to configure the level and appeners
at once, so something like
'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
Thanks.
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Gary Gregory <ga...@gmail.com>.
I agree, let's make it easier for Hadoop :-)
Gary
On Sun, Jan 9, 2022, 15:54 Matt Sicker <bo...@gmail.com> wrote:
> This sounds fairly reasonable, though I haven’t looked at implementation
> details. We should make a Jira ticket to track the change. Otherwise, I
> think it’s a good idea to support the syntax.
>
> —
> Matt Sicker
>
> > On Jan 9, 2022, at 09:02, 张铎 <pa...@gmail.com> wrote:
> >
> > I brought this up in the incubator mailing list, and was suggested to
> > report directly to the log4j community.
> >
> > https://issues.apache.org/jira/browse/HADOOP-16206
> >
> > In hadoop we started to try migrating to log4j2 long ago, but it is not
> > easy. For now, one of the most blocker issues is the lack of support of
> > 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to
> stay
> > on the bridge api, we want to fully migrate to log4j2 finally, so
> > supporting the above grammar in log4j1 bridge is not enough.
> >
> > This is the main log4j configuration file for hadoop
> >
> >
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> >
> > We have this in the file
> >
> > # Define the root logger to the system property "hadoop.root.logger".
> >> log4j.rootLogger=${hadoop.root.logger}
> >
> >
> > So our users could simply pass -Dhadoop.root.logger to control the level
> > and appender.
> >
> >
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> >
> > Here we use an environment variable in our start up scripts. And I
> believe
> > there are lots of other hadoop deployment systems which will have their
> own
> > start scripts.
> >
> > So in general, it is just impossible for us to drop the
> > -Dhadoop.root.logger way of configuring our logging system.
> >
> > So here I want to ask if it is possible for log4j2 to still support the
> > 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
> > back with no change, we just need a way to configure the level and
> appeners
> > at once, so something like
> > 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> >
> > Thanks.
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Matt Sicker <bo...@gmail.com>.
This sounds fairly reasonable, though I haven’t looked at implementation details. We should make a Jira ticket to track the change. Otherwise, I think it’s a good idea to support the syntax.
—
Matt Sicker
> On Jan 9, 2022, at 09:02, 张铎 <pa...@gmail.com> wrote:
>
> I brought this up in the incubator mailing list, and was suggested to
> report directly to the log4j community.
>
> https://issues.apache.org/jira/browse/HADOOP-16206
>
> In hadoop we started to try migrating to log4j2 long ago, but it is not
> easy. For now, one of the most blocker issues is the lack of support of
> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to stay
> on the bridge api, we want to fully migrate to log4j2 finally, so
> supporting the above grammar in log4j1 bridge is not enough.
>
> This is the main log4j configuration file for hadoop
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
>
> We have this in the file
>
> # Define the root logger to the system property "hadoop.root.logger".
>> log4j.rootLogger=${hadoop.root.logger}
>
>
> So our users could simply pass -Dhadoop.root.logger to control the level
> and appender.
>
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
>
> Here we use an environment variable in our start up scripts. And I believe
> there are lots of other hadoop deployment systems which will have their own
> start scripts.
>
> So in general, it is just impossible for us to drop the
> -Dhadoop.root.logger way of configuring our logging system.
>
> So here I want to ask if it is possible for log4j2 to still support the
> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
> back with no change, we just need a way to configure the level and appeners
> at once, so something like
> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
>
> Thanks.
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Dan Kegel <da...@kegel.com>.
Ralph Goers <ra...@dslextreme.com> schrieb am So., 9. Jan. 2022,
20:53:
> StrSubstitutor is already complicated
Having just reviewed that function and its looping caller, I would have to
agree
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
Thanks Ralph, what you proposed is fine by me.
I'm not very familiar with the implementations inside log4j2 so in the
above example I just want to make clear that enhancing property parsing
could also be a direction for solve the problem.
Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 12:53写道：
> OK, I had a suspicion the reason for needing this was something like that.
>
> The syntax you propose below would require modifying PropertiesUtil, the
> Property
> plugin and StrSubstitutor to support arrays. I don’t think I’d want to go
> down that path.
> StrSubstitutor is already complicated and the idea of enhancing it to
> support stuff like
> this doesn’t excite me.
>
> I think we’d be better off creating a new plugin for this. Something like:
>
> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
> The only problem here is if you want to provide more than one appender
> this won’t
> work well. But I am sure we could enhance AppenderRef to accept a list of
> appenders.
>
> Ralph
>
>
>
>
> > On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> > Thank you for the reply.
> >
> > And yes, it is not only for the root logger, we have tons of different
> > loggers in hadoop, as hadoop is constructed with several different
> projects
> > actually, and they are developed by different groups of people...
> >
> > And on splitting the config in shell, actually this is exactly what I
> have
> > done in HBase
> >
> > See
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
> >
> > And also
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
> >
> > For HBase maybe it is acceptable but hadoop is another story.
> > Hadoop is widely used almost in every bigdata platform. Among the
> > companies I've worked with, they all have their own hadoop deployment
> > systems which have modified version of start up scripts...
> >
> > I can imagine that if we do the same trick in HBase, then people will ask
> > again and again on the mailing list why passing -Dhadoop.root.logger does
> > not work, as well as other options like -Dyarn.root.logger,
> > -Dhadoop.security.logger, etc...
> >
> > So it will be good if we can support configure level and appenders in one
> > property.
> >
> > Or maybe another way is to enhance the ability in the Properties section,
> > so we can split one property into two properties, something like
> >
> > <Properties>
> > <Property name="hadoop.root.logger"
> > type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
> > <Property
> > name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
> > <Property name="hadoop.root.logger.appender"> ${hadoop.root.logger[1]}
> > </Property>
> > <Properties>
> >
> > Notice that since we could add multiple appenders here, maybe we need to
> > support something like ${hadoop.root.logger[1:] to combine all the
> > appenders...
> > This could also solve the problem as people could still pass
> > -Dhadoop.root.logger.
> >
> > But I'm not sure if adding the ability to split a string in configuation
> > could introduce new possible security concerns...
> >
> > Thanks.
> >
> > Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
> >
> >> We already support this with two variables. But if they only want to
> pass
> >> one then we have two options:
> >> 1. Modify PropertiesConfiguration to support a new element that allows a
> >> Level and AppenderRef which then gets split internally.
> >> 2. Create a Lookup that extracts the relevant portion of the data.
> >>
> >> Ralph
> >>
> >>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <ga...@gmail.com>
> wrote:
> >>>
> >>> I think it is reasonable to say we can support this through 2 instead
> of
> >> 1
> >>> variable.
> >>>
> >>> Duo?
> >>>
> >>> Gary
> >>>
> >>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <ra...@dslextreme.com>
> >> wrote:
> >>>
> >>>> I’m looking at this and have a couple of concerns. The script has
> >>>>
> >>>>
> >>>> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
> >>>>
> >>>>
> >>
> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
> >>>> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
> >>>>
> >>>> So it seems you need this for more than just the root logger.
> >>>>
> >>>> Second, you are asking us to accept “level, appender” as the value
> and
> >>>> under the covers split the
> >>>> value and assign the level to the level attribute and the appender
> name
> >> to
> >>>> the AppenderRef.
> >>>>
> >>>> While this certainly can be done it seems like it would be just as
> easy
> >> to
> >>>> do the split in the script
> >>>> and create two different variables for the logging configuration to
> pick
> >>>> up. Is there a reason that
> >>>> you can’t do this - for example perhaps users have hacked your scripts
> >> and
> >>>> now you can’t
> >>>> change them without breaking them? If so, was this “supported”?
> >>>>
> >>>> Ralph
> >>>>
> >>>>
> >>>>
> >>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >>>>>
> >>>>> I brought this up in the incubator mailing list, and was suggested to
> >>>>> report directly to the log4j community.
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/HADOOP-16206
> >>>>>
> >>>>> In hadoop we started to try migrating to log4j2 long ago, but it is
> not
> >>>>> easy. For now, one of the most blocker issues is the lack of support
> of
> >>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want
> to
> >>>> stay
> >>>>> on the bridge api, we want to fully migrate to log4j2 finally, so
> >>>>> supporting the above grammar in log4j1 bridge is not enough.
> >>>>>
> >>>>> This is the main log4j configuration file for hadoop
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> >>>>>
> >>>>> We have this in the file
> >>>>>
> >>>>> # Define the root logger to the system property "hadoop.root.logger".
> >>>>>> log4j.rootLogger=${hadoop.root.logger}
> >>>>>
> >>>>>
> >>>>> So our users could simply pass -Dhadoop.root.logger to control the
> >> level
> >>>>> and appender.
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> >>>>>
> >>>>> Here we use an environment variable in our start up scripts. And I
> >>>> believe
> >>>>> there are lots of other hadoop deployment systems which will have
> their
> >>>> own
> >>>>> start scripts.
> >>>>>
> >>>>> So in general, it is just impossible for us to drop the
> >>>>> -Dhadoop.root.logger way of configuring our logging system.
> >>>>>
> >>>>> So here I want to ask if it is possible for log4j2 to still support
> the
> >>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add
> it
> >>>>> back with no change, we just need a way to configure the level and
> >>>> appeners
> >>>>> at once, so something like
> >>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> >>>>>
> >>>>> Thanks.
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
I tried this feature in HBase but got some problems.
I've already filed LOG4J2-3394 to describe the problems.
PTAL. Thanks.
张铎(Duo Zhang) <pa...@gmail.com> 于2022年1月17日周一 12:30写道：
> I spent some time reading the code of log4j2-core, and posted what I have
> learned so far on the jira issue.
>
> I could give it a try but I need a mentor.
>
> Appreciate if someone in the community could help here.
>
> Thanks.
>
> 张铎(Duo Zhang) <pa...@gmail.com> 于2022年1月15日周六 11:54写道：
>
>> Filed https://issues.apache.org/jira/browse/LOG4J2-3341.
>>
>> Gary Gregory <ga...@gmail.com> 于2022年1月11日周二 21:21写道：
>>
>>> On Tue, Jan 11, 2022 at 8:20 AM 张铎(Duo Zhang) <pa...@gmail.com>
>>> wrote:
>>>
>>> > So we could open an issue for this feature then? As we have already
>>> started
>>> > to discuss the implementation :)
>>> >
>>>
>>> Yes, that sounds like a good idea. Feel free to go ahead :-)
>>>
>>> Gary
>>>
>>>
>>> >
>>> > Thanks~
>>> >
>>> > Volkan Yazıcı <vo...@yazi.ci> 于2022年1月11日周二 19:42写道：
>>> >
>>> > > Giving it a one more try:
>>> > >
>>> > > <CompoundProperty>
>>> > > <Source>${sys:hadoop.root.logger}</Source>
>>> > > <Split delimiter="\s+" trim="true" blanksExcluded="true">
>>> > > <item index="0">hadoop.root.logger.level</item>
>>> > > <slice startIndex="1">hadoop.root.logger.appender</slice>
>>> > > </Split>
>>> > > </CompoundProperty>
>>> > >
>>> > > On Mon, Jan 10, 2022 at 3:54 PM Ralph Goers <
>>> ralph.goers@dslextreme.com>
>>> > > wrote:
>>> > >
>>> > > > Yeah, that syntax is better although it doesn’t solve the multiple
>>> > > > appender refs case either.
>>> > > >
>>> > > > LoggerConfig accepts a List<AppenderRef> so it would seem that if
>>> we
>>> > > > modify LoggerConfig
>>> > > > to add another attribute that is a string that creates an
>>> AppenderRef
>>> > for
>>> > > > each item then
>>> > > > one could do
>>> > > >
>>> > > > <Logger name=“xyzzy” level=“INFO” appenderRefNames=“a, b, c”/>
>>> > > >
>>> > > >
>>> > > > Ralph
>>> > > >
>>> > > > > On Jan 10, 2022, at 5:04 AM, Volkan Yazıcı <vo...@yazi.ci>
>>> wrote:
>>> > > > >
>>> > > > > I am curious if shall we adopt a more generic approach to
>>> extraction:
>>> > > > >
>>> > > > > <CompoundProperty>
>>> > > > > <Source>${sys:hadoop.root.logger}</Source>
>>> > > > > <Targets pattern="^(.+)\s*,\s(.+)$">
>>> > > > > <Target>hadoop.root.logger.level</Target>
>>> > > > > <Target>hadoop.root.logger.appender</Target>
>>> > > > > </Targets>
>>> > > > > </CompoundProperty>
>>> > > > >
>>> > > > > On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers <
>>> > > ralph.goers@dslextreme.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > >> OK, I had a suspicion the reason for needing this was something
>>> like
>>> > > > that.
>>> > > > >>
>>> > > > >> The syntax you propose below would require modifying
>>> PropertiesUtil,
>>> > > the
>>> > > > >> Property
>>> > > > >> plugin and StrSubstitutor to support arrays. I don’t think I’d
>>> want
>>> > to
>>> > > > go
>>> > > > >> down that path.
>>> > > > >> StrSubstitutor is already complicated and the idea of enhancing
>>> it
>>> > to
>>> > > > >> support stuff like
>>> > > > >> this doesn’t excite me.
>>> > > > >>
>>> > > > >> I think we’d be better off creating a new plugin for this.
>>> Something
>>> > > > like:
>>> > > > >>
>>> > > > >> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
>>> > > > >> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
>>> > > > >> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
>>> > > > >> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
>>> > > > >>
>>> > > > >> The only problem here is if you want to provide more than one
>>> > appender
>>> > > > >> this won’t
>>> > > > >> work well. But I am sure we could enhance AppenderRef to accept
>>> a
>>> > list
>>> > > > of
>>> > > > >> appenders.
>>> > > > >>
>>> > > > >> Ralph
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >>> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <
>>> palomino219@gmail.com>
>>> > > > wrote:
>>> > > > >>>
>>> > > > >>> Thank you for the reply.
>>> > > > >>>
>>> > > > >>> And yes, it is not only for the root logger, we have tons of
>>> > > different
>>> > > > >>> loggers in hadoop, as hadoop is constructed with several
>>> different
>>> > > > >> projects
>>> > > > >>> actually, and they are developed by different groups of
>>> people...
>>> > > > >>>
>>> > > > >>> And on splitting the config in shell, actually this is exactly
>>> > what I
>>> > > > >> have
>>> > > > >>> done in HBase
>>> > > > >>>
>>> > > > >>> See
>>> > > > >>>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
>>> > > > >>>
>>> > > > >>> And also
>>> > > > >>>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
>>> > > > >>>
>>> > > > >>> For HBase maybe it is acceptable but hadoop is another story.
>>> > > > >>> Hadoop is widely used almost in every bigdata platform. Among
>>> the
>>> > > > >>> companies I've worked with, they all have their own hadoop
>>> > deployment
>>> > > > >>> systems which have modified version of start up scripts...
>>> > > > >>>
>>> > > > >>> I can imagine that if we do the same trick in HBase, then
>>> people
>>> > will
>>> > > > ask
>>> > > > >>> again and again on the mailing list why passing
>>> > -Dhadoop.root.logger
>>> > > > does
>>> > > > >>> not work, as well as other options like -Dyarn.root.logger,
>>> > > > >>> -Dhadoop.security.logger, etc...
>>> > > > >>>
>>> > > > >>> So it will be good if we can support configure level and
>>> appenders
>>> > in
>>> > > > one
>>> > > > >>> property.
>>> > > > >>>
>>> > > > >>> Or maybe another way is to enhance the ability in the
>>> Properties
>>> > > > section,
>>> > > > >>> so we can split one property into two properties, something
>>> like
>>> > > > >>>
>>> > > > >>> <Properties>
>>> > > > >>> <Property name="hadoop.root.logger"
>>> > > > >>> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
>>> > > > >>> <Property
>>> > > > >>>
>>> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
>>> > > > >>> <Property name="hadoop.root.logger.appender">
>>> > > ${hadoop.root.logger[1]}
>>> > > > >>> </Property>
>>> > > > >>> <Properties>
>>> > > > >>>
>>> > > > >>> Notice that since we could add multiple appenders here, maybe
>>> we
>>> > need
>>> > > > to
>>> > > > >>> support something like ${hadoop.root.logger[1:] to combine all
>>> the
>>> > > > >>> appenders...
>>> > > > >>> This could also solve the problem as people could still pass
>>> > > > >>> -Dhadoop.root.logger.
>>> > > > >>>
>>> > > > >>> But I'm not sure if adding the ability to split a string in
>>> > > > configuation
>>> > > > >>> could introduce new possible security concerns...
>>> > > > >>>
>>> > > > >>> Thanks.
>>> > > > >>>
>>> > > > >>> Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一
>>> 07:14写道：
>>> > > > >>>
>>> > > > >>>> We already support this with two variables. But if they only
>>> want
>>> > to
>>> > > > >> pass
>>> > > > >>>> one then we have two options:
>>> > > > >>>> 1. Modify PropertiesConfiguration to support a new element
>>> that
>>> > > > allows a
>>> > > > >>>> Level and AppenderRef which then gets split internally.
>>> > > > >>>> 2. Create a Lookup that extracts the relevant portion of the
>>> data.
>>> > > > >>>>
>>> > > > >>>> Ralph
>>> > > > >>>>
>>> > > > >>>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <
>>> garydgregory@gmail.com
>>> > >
>>> > > > >> wrote:
>>> > > > >>>>>
>>> > > > >>>>> I think it is reasonable to say we can support this through 2
>>> > > instead
>>> > > > >> of
>>> > > > >>>> 1
>>> > > > >>>>> variable.
>>> > > > >>>>>
>>> > > > >>>>> Duo?
>>> > > > >>>>>
>>> > > > >>>>> Gary
>>> > > > >>>>>
>>> > > > >>>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <
>>> > ralph.goers@dslextreme.com
>>> > > >
>>> > > > >>>> wrote:
>>> > > > >>>>>
>>> > > > >>>>>> I’m looking at this and have a couple of concerns. The
>>> script
>>> > has
>>> > > > >>>>>>
>>> > > > >>>>>>
>>> > > > >>>>>>
>>> > > HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
>>> > > > >>>>>>
>>> > > > >>>>>>
>>> > > > >>>>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
>>> > > > >>>>>>
>>> > > HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
>>> > > > >>>>>>
>>> > > > >>>>>> So it seems you need this for more than just the root
>>> logger.
>>> > > > >>>>>>
>>> > > > >>>>>> Second, you are asking us to accept “level, appender” as
>>> the
>>> > > value
>>> > > > >> and
>>> > > > >>>>>> under the covers split the
>>> > > > >>>>>> value and assign the level to the level attribute and the
>>> > appender
>>> > > > >> name
>>> > > > >>>> to
>>> > > > >>>>>> the AppenderRef.
>>> > > > >>>>>>
>>> > > > >>>>>> While this certainly can be done it seems like it would be
>>> just
>>> > as
>>> > > > >> easy
>>> > > > >>>> to
>>> > > > >>>>>> do the split in the script
>>> > > > >>>>>> and create two different variables for the logging
>>> configuration
>>> > > to
>>> > > > >> pick
>>> > > > >>>>>> up. Is there a reason that
>>> > > > >>>>>> you can’t do this - for example perhaps users have hacked
>>> your
>>> > > > scripts
>>> > > > >>>> and
>>> > > > >>>>>> now you can’t
>>> > > > >>>>>> change them without breaking them? If so, was this
>>> “supported”?
>>> > > > >>>>>>
>>> > > > >>>>>> Ralph
>>> > > > >>>>>>
>>> > > > >>>>>>
>>> > > > >>>>>>
>>> > > > >>>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <
>>> > palomino219@gmail.com
>>> > > >
>>> > > > >>>> wrote:
>>> > > > >>>>>>>
>>> > > > >>>>>>> I brought this up in the incubator mailing list, and was
>>> > > suggested
>>> > > > to
>>> > > > >>>>>>> report directly to the log4j community.
>>> > > > >>>>>>>
>>> > > > >>>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
>>> > > > >>>>>>>
>>> > > > >>>>>>> In hadoop we started to try migrating to log4j2 long ago,
>>> but
>>> > it
>>> > > is
>>> > > > >> not
>>> > > > >>>>>>> easy. For now, one of the most blocker issues is the lack
>>> of
>>> > > > support
>>> > > > >> of
>>> > > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do
>>> not
>>> > > want
>>> > > > >> to
>>> > > > >>>>>> stay
>>> > > > >>>>>>> on the bridge api, we want to fully migrate to log4j2
>>> finally,
>>> > so
>>> > > > >>>>>>> supporting the above grammar in log4j1 bridge is not
>>> enough.
>>> > > > >>>>>>>
>>> > > > >>>>>>> This is the main log4j configuration file for hadoop
>>> > > > >>>>>>>
>>> > > > >>>>>>>
>>> > > > >>>>>>
>>> > > > >>>>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
>>> > > > >>>>>>>
>>> > > > >>>>>>> We have this in the file
>>> > > > >>>>>>>
>>> > > > >>>>>>> # Define the root logger to the system property
>>> > > > "hadoop.root.logger".
>>> > > > >>>>>>>> log4j.rootLogger=${hadoop.root.logger}
>>> > > > >>>>>>>
>>> > > > >>>>>>>
>>> > > > >>>>>>> So our users could simply pass -Dhadoop.root.logger to
>>> control
>>> > > the
>>> > > > >>>> level
>>> > > > >>>>>>> and appender.
>>> > > > >>>>>>>
>>> > > > >>>>>>>
>>> > > > >>>>>>
>>> > > > >>>>
>>> > > > >>
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
>>> > > > >>>>>>>
>>> > > > >>>>>>> Here we use an environment variable in our start up
>>> scripts.
>>> > And
>>> > > I
>>> > > > >>>>>> believe
>>> > > > >>>>>>> there are lots of other hadoop deployment systems which
>>> will
>>> > have
>>> > > > >> their
>>> > > > >>>>>> own
>>> > > > >>>>>>> start scripts.
>>> > > > >>>>>>>
>>> > > > >>>>>>> So in general, it is just impossible for us to drop the
>>> > > > >>>>>>> -Dhadoop.root.logger way of configuring our logging system.
>>> > > > >>>>>>>
>>> > > > >>>>>>> So here I want to ask if it is possible for log4j2 to still
>>> > > support
>>> > > > >> the
>>> > > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you
>>> > must
>>> > > > add
>>> > > > >> it
>>> > > > >>>>>>> back with no change, we just need a way to configure the
>>> level
>>> > > and
>>> > > > >>>>>> appeners
>>> > > > >>>>>>> at once, so something like
>>> > > > >>>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also
>>> OK.
>>> > > > >>>>>>>
>>> > > > >>>>>>> Thanks.
>>> > > > >>>>>>
>>> > > > >>>>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>
>>> > > > >>
>>> > > >
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
I spent some time reading the code of log4j2-core, and posted what I have
learned so far on the jira issue.
I could give it a try but I need a mentor.
Appreciate if someone in the community could help here.
Thanks.
张铎(Duo Zhang) <pa...@gmail.com> 于2022年1月15日周六 11:54写道：
> Filed https://issues.apache.org/jira/browse/LOG4J2-3341.
>
> Gary Gregory <ga...@gmail.com> 于2022年1月11日周二 21:21写道：
>
>> On Tue, Jan 11, 2022 at 8:20 AM 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>>
>> > So we could open an issue for this feature then? As we have already
>> started
>> > to discuss the implementation :)
>> >
>>
>> Yes, that sounds like a good idea. Feel free to go ahead :-)
>>
>> Gary
>>
>>
>> >
>> > Thanks~
>> >
>> > Volkan Yazıcı <vo...@yazi.ci> 于2022年1月11日周二 19:42写道：
>> >
>> > > Giving it a one more try:
>> > >
>> > > <CompoundProperty>
>> > > <Source>${sys:hadoop.root.logger}</Source>
>> > > <Split delimiter="\s+" trim="true" blanksExcluded="true">
>> > > <item index="0">hadoop.root.logger.level</item>
>> > > <slice startIndex="1">hadoop.root.logger.appender</slice>
>> > > </Split>
>> > > </CompoundProperty>
>> > >
>> > > On Mon, Jan 10, 2022 at 3:54 PM Ralph Goers <
>> ralph.goers@dslextreme.com>
>> > > wrote:
>> > >
>> > > > Yeah, that syntax is better although it doesn’t solve the multiple
>> > > > appender refs case either.
>> > > >
>> > > > LoggerConfig accepts a List<AppenderRef> so it would seem that if we
>> > > > modify LoggerConfig
>> > > > to add another attribute that is a string that creates an
>> AppenderRef
>> > for
>> > > > each item then
>> > > > one could do
>> > > >
>> > > > <Logger name=“xyzzy” level=“INFO” appenderRefNames=“a, b, c”/>
>> > > >
>> > > >
>> > > > Ralph
>> > > >
>> > > > > On Jan 10, 2022, at 5:04 AM, Volkan Yazıcı <vo...@yazi.ci>
>> wrote:
>> > > > >
>> > > > > I am curious if shall we adopt a more generic approach to
>> extraction:
>> > > > >
>> > > > > <CompoundProperty>
>> > > > > <Source>${sys:hadoop.root.logger}</Source>
>> > > > > <Targets pattern="^(.+)\s*,\s(.+)$">
>> > > > > <Target>hadoop.root.logger.level</Target>
>> > > > > <Target>hadoop.root.logger.appender</Target>
>> > > > > </Targets>
>> > > > > </CompoundProperty>
>> > > > >
>> > > > > On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers <
>> > > ralph.goers@dslextreme.com>
>> > > > > wrote:
>> > > > >
>> > > > >> OK, I had a suspicion the reason for needing this was something
>> like
>> > > > that.
>> > > > >>
>> > > > >> The syntax you propose below would require modifying
>> PropertiesUtil,
>> > > the
>> > > > >> Property
>> > > > >> plugin and StrSubstitutor to support arrays. I don’t think I’d
>> want
>> > to
>> > > > go
>> > > > >> down that path.
>> > > > >> StrSubstitutor is already complicated and the idea of enhancing
>> it
>> > to
>> > > > >> support stuff like
>> > > > >> this doesn’t excite me.
>> > > > >>
>> > > > >> I think we’d be better off creating a new plugin for this.
>> Something
>> > > > like:
>> > > > >>
>> > > > >> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
>> > > > >> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
>> > > > >> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
>> > > > >> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
>> > > > >>
>> > > > >> The only problem here is if you want to provide more than one
>> > appender
>> > > > >> this won’t
>> > > > >> work well. But I am sure we could enhance AppenderRef to accept a
>> > list
>> > > > of
>> > > > >> appenders.
>> > > > >>
>> > > > >> Ralph
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >>> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <
>> palomino219@gmail.com>
>> > > > wrote:
>> > > > >>>
>> > > > >>> Thank you for the reply.
>> > > > >>>
>> > > > >>> And yes, it is not only for the root logger, we have tons of
>> > > different
>> > > > >>> loggers in hadoop, as hadoop is constructed with several
>> different
>> > > > >> projects
>> > > > >>> actually, and they are developed by different groups of
>> people...
>> > > > >>>
>> > > > >>> And on splitting the config in shell, actually this is exactly
>> > what I
>> > > > >> have
>> > > > >>> done in HBase
>> > > > >>>
>> > > > >>> See
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
>> > > > >>>
>> > > > >>> And also
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
>> > > > >>>
>> > > > >>> For HBase maybe it is acceptable but hadoop is another story.
>> > > > >>> Hadoop is widely used almost in every bigdata platform. Among
>> the
>> > > > >>> companies I've worked with, they all have their own hadoop
>> > deployment
>> > > > >>> systems which have modified version of start up scripts...
>> > > > >>>
>> > > > >>> I can imagine that if we do the same trick in HBase, then people
>> > will
>> > > > ask
>> > > > >>> again and again on the mailing list why passing
>> > -Dhadoop.root.logger
>> > > > does
>> > > > >>> not work, as well as other options like -Dyarn.root.logger,
>> > > > >>> -Dhadoop.security.logger, etc...
>> > > > >>>
>> > > > >>> So it will be good if we can support configure level and
>> appenders
>> > in
>> > > > one
>> > > > >>> property.
>> > > > >>>
>> > > > >>> Or maybe another way is to enhance the ability in the Properties
>> > > > section,
>> > > > >>> so we can split one property into two properties, something like
>> > > > >>>
>> > > > >>> <Properties>
>> > > > >>> <Property name="hadoop.root.logger"
>> > > > >>> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
>> > > > >>> <Property
>> > > > >>>
>> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
>> > > > >>> <Property name="hadoop.root.logger.appender">
>> > > ${hadoop.root.logger[1]}
>> > > > >>> </Property>
>> > > > >>> <Properties>
>> > > > >>>
>> > > > >>> Notice that since we could add multiple appenders here, maybe we
>> > need
>> > > > to
>> > > > >>> support something like ${hadoop.root.logger[1:] to combine all
>> the
>> > > > >>> appenders...
>> > > > >>> This could also solve the problem as people could still pass
>> > > > >>> -Dhadoop.root.logger.
>> > > > >>>
>> > > > >>> But I'm not sure if adding the ability to split a string in
>> > > > configuation
>> > > > >>> could introduce new possible security concerns...
>> > > > >>>
>> > > > >>> Thanks.
>> > > > >>>
>> > > > >>> Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
>> > > > >>>
>> > > > >>>> We already support this with two variables. But if they only
>> want
>> > to
>> > > > >> pass
>> > > > >>>> one then we have two options:
>> > > > >>>> 1. Modify PropertiesConfiguration to support a new element that
>> > > > allows a
>> > > > >>>> Level and AppenderRef which then gets split internally.
>> > > > >>>> 2. Create a Lookup that extracts the relevant portion of the
>> data.
>> > > > >>>>
>> > > > >>>> Ralph
>> > > > >>>>
>> > > > >>>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <
>> garydgregory@gmail.com
>> > >
>> > > > >> wrote:
>> > > > >>>>>
>> > > > >>>>> I think it is reasonable to say we can support this through 2
>> > > instead
>> > > > >> of
>> > > > >>>> 1
>> > > > >>>>> variable.
>> > > > >>>>>
>> > > > >>>>> Duo?
>> > > > >>>>>
>> > > > >>>>> Gary
>> > > > >>>>>
>> > > > >>>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <
>> > ralph.goers@dslextreme.com
>> > > >
>> > > > >>>> wrote:
>> > > > >>>>>
>> > > > >>>>>> I’m looking at this and have a couple of concerns. The script
>> > has
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>>
>> > > HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>
>> > > > >>
>> > > >
>> > >
>> >
>> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
>> > > > >>>>>>
>> > > HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
>> > > > >>>>>>
>> > > > >>>>>> So it seems you need this for more than just the root logger.
>> > > > >>>>>>
>> > > > >>>>>> Second, you are asking us to accept “level, appender” as the
>> > > value
>> > > > >> and
>> > > > >>>>>> under the covers split the
>> > > > >>>>>> value and assign the level to the level attribute and the
>> > appender
>> > > > >> name
>> > > > >>>> to
>> > > > >>>>>> the AppenderRef.
>> > > > >>>>>>
>> > > > >>>>>> While this certainly can be done it seems like it would be
>> just
>> > as
>> > > > >> easy
>> > > > >>>> to
>> > > > >>>>>> do the split in the script
>> > > > >>>>>> and create two different variables for the logging
>> configuration
>> > > to
>> > > > >> pick
>> > > > >>>>>> up. Is there a reason that
>> > > > >>>>>> you can’t do this - for example perhaps users have hacked
>> your
>> > > > scripts
>> > > > >>>> and
>> > > > >>>>>> now you can’t
>> > > > >>>>>> change them without breaking them? If so, was this
>> “supported”?
>> > > > >>>>>>
>> > > > >>>>>> Ralph
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <
>> > palomino219@gmail.com
>> > > >
>> > > > >>>> wrote:
>> > > > >>>>>>>
>> > > > >>>>>>> I brought this up in the incubator mailing list, and was
>> > > suggested
>> > > > to
>> > > > >>>>>>> report directly to the log4j community.
>> > > > >>>>>>>
>> > > > >>>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
>> > > > >>>>>>>
>> > > > >>>>>>> In hadoop we started to try migrating to log4j2 long ago,
>> but
>> > it
>> > > is
>> > > > >> not
>> > > > >>>>>>> easy. For now, one of the most blocker issues is the lack of
>> > > > support
>> > > > >> of
>> > > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do
>> not
>> > > want
>> > > > >> to
>> > > > >>>>>> stay
>> > > > >>>>>>> on the bridge api, we want to fully migrate to log4j2
>> finally,
>> > so
>> > > > >>>>>>> supporting the above grammar in log4j1 bridge is not enough.
>> > > > >>>>>>>
>> > > > >>>>>>> This is the main log4j configuration file for hadoop
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>
>> > > > >>>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
>> > > > >>>>>>>
>> > > > >>>>>>> We have this in the file
>> > > > >>>>>>>
>> > > > >>>>>>> # Define the root logger to the system property
>> > > > "hadoop.root.logger".
>> > > > >>>>>>>> log4j.rootLogger=${hadoop.root.logger}
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>> So our users could simply pass -Dhadoop.root.logger to
>> control
>> > > the
>> > > > >>>> level
>> > > > >>>>>>> and appender.
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>
>> > > > >>>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
>> > > > >>>>>>>
>> > > > >>>>>>> Here we use an environment variable in our start up scripts.
>> > And
>> > > I
>> > > > >>>>>> believe
>> > > > >>>>>>> there are lots of other hadoop deployment systems which will
>> > have
>> > > > >> their
>> > > > >>>>>> own
>> > > > >>>>>>> start scripts.
>> > > > >>>>>>>
>> > > > >>>>>>> So in general, it is just impossible for us to drop the
>> > > > >>>>>>> -Dhadoop.root.logger way of configuring our logging system.
>> > > > >>>>>>>
>> > > > >>>>>>> So here I want to ask if it is possible for log4j2 to still
>> > > support
>> > > > >> the
>> > > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you
>> > must
>> > > > add
>> > > > >> it
>> > > > >>>>>>> back with no change, we just need a way to configure the
>> level
>> > > and
>> > > > >>>>>> appeners
>> > > > >>>>>>> at once, so something like
>> > > > >>>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also
>> OK.
>> > > > >>>>>>>
>> > > > >>>>>>> Thanks.
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>
>> > > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
Filed https://issues.apache.org/jira/browse/LOG4J2-3341.
Gary Gregory <ga...@gmail.com> 于2022年1月11日周二 21:21写道：
> On Tue, Jan 11, 2022 at 8:20 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
> > So we could open an issue for this feature then? As we have already
> started
> > to discuss the implementation :)
> >
>
> Yes, that sounds like a good idea. Feel free to go ahead :-)
>
> Gary
>
>
> >
> > Thanks~
> >
> > Volkan Yazıcı <vo...@yazi.ci> 于2022年1月11日周二 19:42写道：
> >
> > > Giving it a one more try:
> > >
> > > <CompoundProperty>
> > > <Source>${sys:hadoop.root.logger}</Source>
> > > <Split delimiter="\s+" trim="true" blanksExcluded="true">
> > > <item index="0">hadoop.root.logger.level</item>
> > > <slice startIndex="1">hadoop.root.logger.appender</slice>
> > > </Split>
> > > </CompoundProperty>
> > >
> > > On Mon, Jan 10, 2022 at 3:54 PM Ralph Goers <
> ralph.goers@dslextreme.com>
> > > wrote:
> > >
> > > > Yeah, that syntax is better although it doesn’t solve the multiple
> > > > appender refs case either.
> > > >
> > > > LoggerConfig accepts a List<AppenderRef> so it would seem that if we
> > > > modify LoggerConfig
> > > > to add another attribute that is a string that creates an AppenderRef
> > for
> > > > each item then
> > > > one could do
> > > >
> > > > <Logger name=“xyzzy” level=“INFO” appenderRefNames=“a, b, c”/>
> > > >
> > > >
> > > > Ralph
> > > >
> > > > > On Jan 10, 2022, at 5:04 AM, Volkan Yazıcı <vo...@yazi.ci> wrote:
> > > > >
> > > > > I am curious if shall we adopt a more generic approach to
> extraction:
> > > > >
> > > > > <CompoundProperty>
> > > > > <Source>${sys:hadoop.root.logger}</Source>
> > > > > <Targets pattern="^(.+)\s*,\s(.+)$">
> > > > > <Target>hadoop.root.logger.level</Target>
> > > > > <Target>hadoop.root.logger.appender</Target>
> > > > > </Targets>
> > > > > </CompoundProperty>
> > > > >
> > > > > On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers <
> > > ralph.goers@dslextreme.com>
> > > > > wrote:
> > > > >
> > > > >> OK, I had a suspicion the reason for needing this was something
> like
> > > > that.
> > > > >>
> > > > >> The syntax you propose below would require modifying
> PropertiesUtil,
> > > the
> > > > >> Property
> > > > >> plugin and StrSubstitutor to support arrays. I don’t think I’d
> want
> > to
> > > > go
> > > > >> down that path.
> > > > >> StrSubstitutor is already complicated and the idea of enhancing it
> > to
> > > > >> support stuff like
> > > > >> this doesn’t excite me.
> > > > >>
> > > > >> I think we’d be better off creating a new plugin for this.
> Something
> > > > like:
> > > > >>
> > > > >> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
> > > > >> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
> > > > >> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
> > > > >> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
> > > > >>
> > > > >> The only problem here is if you want to provide more than one
> > appender
> > > > >> this won’t
> > > > >> work well. But I am sure we could enhance AppenderRef to accept a
> > list
> > > > of
> > > > >> appenders.
> > > > >>
> > > > >> Ralph
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <palomino219@gmail.com
> >
> > > > wrote:
> > > > >>>
> > > > >>> Thank you for the reply.
> > > > >>>
> > > > >>> And yes, it is not only for the root logger, we have tons of
> > > different
> > > > >>> loggers in hadoop, as hadoop is constructed with several
> different
> > > > >> projects
> > > > >>> actually, and they are developed by different groups of people...
> > > > >>>
> > > > >>> And on splitting the config in shell, actually this is exactly
> > what I
> > > > >> have
> > > > >>> done in HBase
> > > > >>>
> > > > >>> See
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
> > > > >>>
> > > > >>> And also
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
> > > > >>>
> > > > >>> For HBase maybe it is acceptable but hadoop is another story.
> > > > >>> Hadoop is widely used almost in every bigdata platform. Among the
> > > > >>> companies I've worked with, they all have their own hadoop
> > deployment
> > > > >>> systems which have modified version of start up scripts...
> > > > >>>
> > > > >>> I can imagine that if we do the same trick in HBase, then people
> > will
> > > > ask
> > > > >>> again and again on the mailing list why passing
> > -Dhadoop.root.logger
> > > > does
> > > > >>> not work, as well as other options like -Dyarn.root.logger,
> > > > >>> -Dhadoop.security.logger, etc...
> > > > >>>
> > > > >>> So it will be good if we can support configure level and
> appenders
> > in
> > > > one
> > > > >>> property.
> > > > >>>
> > > > >>> Or maybe another way is to enhance the ability in the Properties
> > > > section,
> > > > >>> so we can split one property into two properties, something like
> > > > >>>
> > > > >>> <Properties>
> > > > >>> <Property name="hadoop.root.logger"
> > > > >>> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
> > > > >>> <Property
> > > > >>>
> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
> > > > >>> <Property name="hadoop.root.logger.appender">
> > > ${hadoop.root.logger[1]}
> > > > >>> </Property>
> > > > >>> <Properties>
> > > > >>>
> > > > >>> Notice that since we could add multiple appenders here, maybe we
> > need
> > > > to
> > > > >>> support something like ${hadoop.root.logger[1:] to combine all
> the
> > > > >>> appenders...
> > > > >>> This could also solve the problem as people could still pass
> > > > >>> -Dhadoop.root.logger.
> > > > >>>
> > > > >>> But I'm not sure if adding the ability to split a string in
> > > > configuation
> > > > >>> could introduce new possible security concerns...
> > > > >>>
> > > > >>> Thanks.
> > > > >>>
> > > > >>> Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
> > > > >>>
> > > > >>>> We already support this with two variables. But if they only
> want
> > to
> > > > >> pass
> > > > >>>> one then we have two options:
> > > > >>>> 1. Modify PropertiesConfiguration to support a new element that
> > > > allows a
> > > > >>>> Level and AppenderRef which then gets split internally.
> > > > >>>> 2. Create a Lookup that extracts the relevant portion of the
> data.
> > > > >>>>
> > > > >>>> Ralph
> > > > >>>>
> > > > >>>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <
> garydgregory@gmail.com
> > >
> > > > >> wrote:
> > > > >>>>>
> > > > >>>>> I think it is reasonable to say we can support this through 2
> > > instead
> > > > >> of
> > > > >>>> 1
> > > > >>>>> variable.
> > > > >>>>>
> > > > >>>>> Duo?
> > > > >>>>>
> > > > >>>>> Gary
> > > > >>>>>
> > > > >>>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <
> > ralph.goers@dslextreme.com
> > > >
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> I’m looking at this and have a couple of concerns. The script
> > has
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
> > > > >>>>>>
> > > HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
> > > > >>>>>>
> > > > >>>>>> So it seems you need this for more than just the root logger.
> > > > >>>>>>
> > > > >>>>>> Second, you are asking us to accept “level, appender” as the
> > > value
> > > > >> and
> > > > >>>>>> under the covers split the
> > > > >>>>>> value and assign the level to the level attribute and the
> > appender
> > > > >> name
> > > > >>>> to
> > > > >>>>>> the AppenderRef.
> > > > >>>>>>
> > > > >>>>>> While this certainly can be done it seems like it would be
> just
> > as
> > > > >> easy
> > > > >>>> to
> > > > >>>>>> do the split in the script
> > > > >>>>>> and create two different variables for the logging
> configuration
> > > to
> > > > >> pick
> > > > >>>>>> up. Is there a reason that
> > > > >>>>>> you can’t do this - for example perhaps users have hacked your
> > > > scripts
> > > > >>>> and
> > > > >>>>>> now you can’t
> > > > >>>>>> change them without breaking them? If so, was this
> “supported”?
> > > > >>>>>>
> > > > >>>>>> Ralph
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <
> > palomino219@gmail.com
> > > >
> > > > >>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>> I brought this up in the incubator mailing list, and was
> > > suggested
> > > > to
> > > > >>>>>>> report directly to the log4j community.
> > > > >>>>>>>
> > > > >>>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
> > > > >>>>>>>
> > > > >>>>>>> In hadoop we started to try migrating to log4j2 long ago, but
> > it
> > > is
> > > > >> not
> > > > >>>>>>> easy. For now, one of the most blocker issues is the lack of
> > > > support
> > > > >> of
> > > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do
> not
> > > want
> > > > >> to
> > > > >>>>>> stay
> > > > >>>>>>> on the bridge api, we want to fully migrate to log4j2
> finally,
> > so
> > > > >>>>>>> supporting the above grammar in log4j1 bridge is not enough.
> > > > >>>>>>>
> > > > >>>>>>> This is the main log4j configuration file for hadoop
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> > > > >>>>>>>
> > > > >>>>>>> We have this in the file
> > > > >>>>>>>
> > > > >>>>>>> # Define the root logger to the system property
> > > > "hadoop.root.logger".
> > > > >>>>>>>> log4j.rootLogger=${hadoop.root.logger}
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> So our users could simply pass -Dhadoop.root.logger to
> control
> > > the
> > > > >>>> level
> > > > >>>>>>> and appender.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> > > > >>>>>>>
> > > > >>>>>>> Here we use an environment variable in our start up scripts.
> > And
> > > I
> > > > >>>>>> believe
> > > > >>>>>>> there are lots of other hadoop deployment systems which will
> > have
> > > > >> their
> > > > >>>>>> own
> > > > >>>>>>> start scripts.
> > > > >>>>>>>
> > > > >>>>>>> So in general, it is just impossible for us to drop the
> > > > >>>>>>> -Dhadoop.root.logger way of configuring our logging system.
> > > > >>>>>>>
> > > > >>>>>>> So here I want to ask if it is possible for log4j2 to still
> > > support
> > > > >> the
> > > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you
> > must
> > > > add
> > > > >> it
> > > > >>>>>>> back with no change, we just need a way to configure the
> level
> > > and
> > > > >>>>>> appeners
> > > > >>>>>>> at once, so something like
> > > > >>>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> > > > >>>>>>>
> > > > >>>>>>> Thanks.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Gary Gregory <ga...@gmail.com>.
On Tue, Jan 11, 2022 at 8:20 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> So we could open an issue for this feature then? As we have already started
> to discuss the implementation :)
>
Yes, that sounds like a good idea. Feel free to go ahead :-)
Gary
>
> Thanks~
>
> Volkan Yazıcı <vo...@yazi.ci> 于2022年1月11日周二 19:42写道：
>
> > Giving it a one more try:
> >
> > <CompoundProperty>
> > <Source>${sys:hadoop.root.logger}</Source>
> > <Split delimiter="\s+" trim="true" blanksExcluded="true">
> > <item index="0">hadoop.root.logger.level</item>
> > <slice startIndex="1">hadoop.root.logger.appender</slice>
> > </Split>
> > </CompoundProperty>
> >
> > On Mon, Jan 10, 2022 at 3:54 PM Ralph Goers <ra...@dslextreme.com>
> > wrote:
> >
> > > Yeah, that syntax is better although it doesn’t solve the multiple
> > > appender refs case either.
> > >
> > > LoggerConfig accepts a List<AppenderRef> so it would seem that if we
> > > modify LoggerConfig
> > > to add another attribute that is a string that creates an AppenderRef
> for
> > > each item then
> > > one could do
> > >
> > > <Logger name=“xyzzy” level=“INFO” appenderRefNames=“a, b, c”/>
> > >
> > >
> > > Ralph
> > >
> > > > On Jan 10, 2022, at 5:04 AM, Volkan Yazıcı <vo...@yazi.ci> wrote:
> > > >
> > > > I am curious if shall we adopt a more generic approach to extraction:
> > > >
> > > > <CompoundProperty>
> > > > <Source>${sys:hadoop.root.logger}</Source>
> > > > <Targets pattern="^(.+)\s*,\s(.+)$">
> > > > <Target>hadoop.root.logger.level</Target>
> > > > <Target>hadoop.root.logger.appender</Target>
> > > > </Targets>
> > > > </CompoundProperty>
> > > >
> > > > On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers <
> > ralph.goers@dslextreme.com>
> > > > wrote:
> > > >
> > > >> OK, I had a suspicion the reason for needing this was something like
> > > that.
> > > >>
> > > >> The syntax you propose below would require modifying PropertiesUtil,
> > the
> > > >> Property
> > > >> plugin and StrSubstitutor to support arrays. I don’t think I’d want
> to
> > > go
> > > >> down that path.
> > > >> StrSubstitutor is already complicated and the idea of enhancing it
> to
> > > >> support stuff like
> > > >> this doesn’t excite me.
> > > >>
> > > >> I think we’d be better off creating a new plugin for this. Something
> > > like:
> > > >>
> > > >> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
> > > >> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
> > > >> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
> > > >> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
> > > >>
> > > >> The only problem here is if you want to provide more than one
> appender
> > > >> this won’t
> > > >> work well. But I am sure we could enhance AppenderRef to accept a
> list
> > > of
> > > >> appenders.
> > > >>
> > > >> Ralph
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <pa...@gmail.com>
> > > wrote:
> > > >>>
> > > >>> Thank you for the reply.
> > > >>>
> > > >>> And yes, it is not only for the root logger, we have tons of
> > different
> > > >>> loggers in hadoop, as hadoop is constructed with several different
> > > >> projects
> > > >>> actually, and they are developed by different groups of people...
> > > >>>
> > > >>> And on splitting the config in shell, actually this is exactly
> what I
> > > >> have
> > > >>> done in HBase
> > > >>>
> > > >>> See
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
> > > >>>
> > > >>> And also
> > > >>>
> > > >>
> > >
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
> > > >>>
> > > >>> For HBase maybe it is acceptable but hadoop is another story.
> > > >>> Hadoop is widely used almost in every bigdata platform. Among the
> > > >>> companies I've worked with, they all have their own hadoop
> deployment
> > > >>> systems which have modified version of start up scripts...
> > > >>>
> > > >>> I can imagine that if we do the same trick in HBase, then people
> will
> > > ask
> > > >>> again and again on the mailing list why passing
> -Dhadoop.root.logger
> > > does
> > > >>> not work, as well as other options like -Dyarn.root.logger,
> > > >>> -Dhadoop.security.logger, etc...
> > > >>>
> > > >>> So it will be good if we can support configure level and appenders
> in
> > > one
> > > >>> property.
> > > >>>
> > > >>> Or maybe another way is to enhance the ability in the Properties
> > > section,
> > > >>> so we can split one property into two properties, something like
> > > >>>
> > > >>> <Properties>
> > > >>> <Property name="hadoop.root.logger"
> > > >>> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
> > > >>> <Property
> > > >>> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
> > > >>> <Property name="hadoop.root.logger.appender">
> > ${hadoop.root.logger[1]}
> > > >>> </Property>
> > > >>> <Properties>
> > > >>>
> > > >>> Notice that since we could add multiple appenders here, maybe we
> need
> > > to
> > > >>> support something like ${hadoop.root.logger[1:] to combine all the
> > > >>> appenders...
> > > >>> This could also solve the problem as people could still pass
> > > >>> -Dhadoop.root.logger.
> > > >>>
> > > >>> But I'm not sure if adding the ability to split a string in
> > > configuation
> > > >>> could introduce new possible security concerns...
> > > >>>
> > > >>> Thanks.
> > > >>>
> > > >>> Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
> > > >>>
> > > >>>> We already support this with two variables. But if they only want
> to
> > > >> pass
> > > >>>> one then we have two options:
> > > >>>> 1. Modify PropertiesConfiguration to support a new element that
> > > allows a
> > > >>>> Level and AppenderRef which then gets split internally.
> > > >>>> 2. Create a Lookup that extracts the relevant portion of the data.
> > > >>>>
> > > >>>> Ralph
> > > >>>>
> > > >>>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <garydgregory@gmail.com
> >
> > > >> wrote:
> > > >>>>>
> > > >>>>> I think it is reasonable to say we can support this through 2
> > instead
> > > >> of
> > > >>>> 1
> > > >>>>> variable.
> > > >>>>>
> > > >>>>> Duo?
> > > >>>>>
> > > >>>>> Gary
> > > >>>>>
> > > >>>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <
> ralph.goers@dslextreme.com
> > >
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> I’m looking at this and have a couple of concerns. The script
> has
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
> > > >>>>>>
> > HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
> > > >>>>>>
> > > >>>>>> So it seems you need this for more than just the root logger.
> > > >>>>>>
> > > >>>>>> Second, you are asking us to accept “level, appender” as the
> > value
> > > >> and
> > > >>>>>> under the covers split the
> > > >>>>>> value and assign the level to the level attribute and the
> appender
> > > >> name
> > > >>>> to
> > > >>>>>> the AppenderRef.
> > > >>>>>>
> > > >>>>>> While this certainly can be done it seems like it would be just
> as
> > > >> easy
> > > >>>> to
> > > >>>>>> do the split in the script
> > > >>>>>> and create two different variables for the logging configuration
> > to
> > > >> pick
> > > >>>>>> up. Is there a reason that
> > > >>>>>> you can’t do this - for example perhaps users have hacked your
> > > scripts
> > > >>>> and
> > > >>>>>> now you can’t
> > > >>>>>> change them without breaking them? If so, was this “supported”?
> > > >>>>>>
> > > >>>>>> Ralph
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <
> palomino219@gmail.com
> > >
> > > >>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I brought this up in the incubator mailing list, and was
> > suggested
> > > to
> > > >>>>>>> report directly to the log4j community.
> > > >>>>>>>
> > > >>>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
> > > >>>>>>>
> > > >>>>>>> In hadoop we started to try migrating to log4j2 long ago, but
> it
> > is
> > > >> not
> > > >>>>>>> easy. For now, one of the most blocker issues is the lack of
> > > support
> > > >> of
> > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not
> > want
> > > >> to
> > > >>>>>> stay
> > > >>>>>>> on the bridge api, we want to fully migrate to log4j2 finally,
> so
> > > >>>>>>> supporting the above grammar in log4j1 bridge is not enough.
> > > >>>>>>>
> > > >>>>>>> This is the main log4j configuration file for hadoop
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> > > >>>>>>>
> > > >>>>>>> We have this in the file
> > > >>>>>>>
> > > >>>>>>> # Define the root logger to the system property
> > > "hadoop.root.logger".
> > > >>>>>>>> log4j.rootLogger=${hadoop.root.logger}
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> So our users could simply pass -Dhadoop.root.logger to control
> > the
> > > >>>> level
> > > >>>>>>> and appender.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> > > >>>>>>>
> > > >>>>>>> Here we use an environment variable in our start up scripts.
> And
> > I
> > > >>>>>> believe
> > > >>>>>>> there are lots of other hadoop deployment systems which will
> have
> > > >> their
> > > >>>>>> own
> > > >>>>>>> start scripts.
> > > >>>>>>>
> > > >>>>>>> So in general, it is just impossible for us to drop the
> > > >>>>>>> -Dhadoop.root.logger way of configuring our logging system.
> > > >>>>>>>
> > > >>>>>>> So here I want to ask if it is possible for log4j2 to still
> > support
> > > >> the
> > > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you
> must
> > > add
> > > >> it
> > > >>>>>>> back with no change, we just need a way to configure the level
> > and
> > > >>>>>> appeners
> > > >>>>>>> at once, so something like
> > > >>>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> > > >>>>>>>
> > > >>>>>>> Thanks.
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
So we could open an issue for this feature then? As we have already started
to discuss the implementation :)
Thanks~
Volkan Yazıcı <vo...@yazi.ci> 于2022年1月11日周二 19:42写道：
> Giving it a one more try:
>
> <CompoundProperty>
> <Source>${sys:hadoop.root.logger}</Source>
> <Split delimiter="\s+" trim="true" blanksExcluded="true">
> <item index="0">hadoop.root.logger.level</item>
> <slice startIndex="1">hadoop.root.logger.appender</slice>
> </Split>
> </CompoundProperty>
>
> On Mon, Jan 10, 2022 at 3:54 PM Ralph Goers <ra...@dslextreme.com>
> wrote:
>
> > Yeah, that syntax is better although it doesn’t solve the multiple
> > appender refs case either.
> >
> > LoggerConfig accepts a List<AppenderRef> so it would seem that if we
> > modify LoggerConfig
> > to add another attribute that is a string that creates an AppenderRef for
> > each item then
> > one could do
> >
> > <Logger name=“xyzzy” level=“INFO” appenderRefNames=“a, b, c”/>
> >
> >
> > Ralph
> >
> > > On Jan 10, 2022, at 5:04 AM, Volkan Yazıcı <vo...@yazi.ci> wrote:
> > >
> > > I am curious if shall we adopt a more generic approach to extraction:
> > >
> > > <CompoundProperty>
> > > <Source>${sys:hadoop.root.logger}</Source>
> > > <Targets pattern="^(.+)\s*,\s(.+)$">
> > > <Target>hadoop.root.logger.level</Target>
> > > <Target>hadoop.root.logger.appender</Target>
> > > </Targets>
> > > </CompoundProperty>
> > >
> > > On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers <
> ralph.goers@dslextreme.com>
> > > wrote:
> > >
> > >> OK, I had a suspicion the reason for needing this was something like
> > that.
> > >>
> > >> The syntax you propose below would require modifying PropertiesUtil,
> the
> > >> Property
> > >> plugin and StrSubstitutor to support arrays. I don’t think I’d want to
> > go
> > >> down that path.
> > >> StrSubstitutor is already complicated and the idea of enhancing it to
> > >> support stuff like
> > >> this doesn’t excite me.
> > >>
> > >> I think we’d be better off creating a new plugin for this. Something
> > like:
> > >>
> > >> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
> > >> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
> > >> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
> > >> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
> > >>
> > >> The only problem here is if you want to provide more than one appender
> > >> this won’t
> > >> work well. But I am sure we could enhance AppenderRef to accept a list
> > of
> > >> appenders.
> > >>
> > >> Ralph
> > >>
> > >>
> > >>
> > >>
> > >>> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> > >>>
> > >>> Thank you for the reply.
> > >>>
> > >>> And yes, it is not only for the root logger, we have tons of
> different
> > >>> loggers in hadoop, as hadoop is constructed with several different
> > >> projects
> > >>> actually, and they are developed by different groups of people...
> > >>>
> > >>> And on splitting the config in shell, actually this is exactly what I
> > >> have
> > >>> done in HBase
> > >>>
> > >>> See
> > >>>
> > >>
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
> > >>>
> > >>> And also
> > >>>
> > >>
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
> > >>>
> > >>> For HBase maybe it is acceptable but hadoop is another story.
> > >>> Hadoop is widely used almost in every bigdata platform. Among the
> > >>> companies I've worked with, they all have their own hadoop deployment
> > >>> systems which have modified version of start up scripts...
> > >>>
> > >>> I can imagine that if we do the same trick in HBase, then people will
> > ask
> > >>> again and again on the mailing list why passing -Dhadoop.root.logger
> > does
> > >>> not work, as well as other options like -Dyarn.root.logger,
> > >>> -Dhadoop.security.logger, etc...
> > >>>
> > >>> So it will be good if we can support configure level and appenders in
> > one
> > >>> property.
> > >>>
> > >>> Or maybe another way is to enhance the ability in the Properties
> > section,
> > >>> so we can split one property into two properties, something like
> > >>>
> > >>> <Properties>
> > >>> <Property name="hadoop.root.logger"
> > >>> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
> > >>> <Property
> > >>> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
> > >>> <Property name="hadoop.root.logger.appender">
> ${hadoop.root.logger[1]}
> > >>> </Property>
> > >>> <Properties>
> > >>>
> > >>> Notice that since we could add multiple appenders here, maybe we need
> > to
> > >>> support something like ${hadoop.root.logger[1:] to combine all the
> > >>> appenders...
> > >>> This could also solve the problem as people could still pass
> > >>> -Dhadoop.root.logger.
> > >>>
> > >>> But I'm not sure if adding the ability to split a string in
> > configuation
> > >>> could introduce new possible security concerns...
> > >>>
> > >>> Thanks.
> > >>>
> > >>> Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
> > >>>
> > >>>> We already support this with two variables. But if they only want to
> > >> pass
> > >>>> one then we have two options:
> > >>>> 1. Modify PropertiesConfiguration to support a new element that
> > allows a
> > >>>> Level and AppenderRef which then gets split internally.
> > >>>> 2. Create a Lookup that extracts the relevant portion of the data.
> > >>>>
> > >>>> Ralph
> > >>>>
> > >>>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <ga...@gmail.com>
> > >> wrote:
> > >>>>>
> > >>>>> I think it is reasonable to say we can support this through 2
> instead
> > >> of
> > >>>> 1
> > >>>>> variable.
> > >>>>>
> > >>>>> Duo?
> > >>>>>
> > >>>>> Gary
> > >>>>>
> > >>>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <ralph.goers@dslextreme.com
> >
> > >>>> wrote:
> > >>>>>
> > >>>>>> I’m looking at this and have a couple of concerns. The script has
> > >>>>>>
> > >>>>>>
> > >>>>>>
> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
> > >>>>>>
> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
> > >>>>>>
> > >>>>>> So it seems you need this for more than just the root logger.
> > >>>>>>
> > >>>>>> Second, you are asking us to accept “level, appender” as the
> value
> > >> and
> > >>>>>> under the covers split the
> > >>>>>> value and assign the level to the level attribute and the appender
> > >> name
> > >>>> to
> > >>>>>> the AppenderRef.
> > >>>>>>
> > >>>>>> While this certainly can be done it seems like it would be just as
> > >> easy
> > >>>> to
> > >>>>>> do the split in the script
> > >>>>>> and create two different variables for the logging configuration
> to
> > >> pick
> > >>>>>> up. Is there a reason that
> > >>>>>> you can’t do this - for example perhaps users have hacked your
> > scripts
> > >>>> and
> > >>>>>> now you can’t
> > >>>>>> change them without breaking them? If so, was this “supported”?
> > >>>>>>
> > >>>>>> Ralph
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <palomino219@gmail.com
> >
> > >>>> wrote:
> > >>>>>>>
> > >>>>>>> I brought this up in the incubator mailing list, and was
> suggested
> > to
> > >>>>>>> report directly to the log4j community.
> > >>>>>>>
> > >>>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
> > >>>>>>>
> > >>>>>>> In hadoop we started to try migrating to log4j2 long ago, but it
> is
> > >> not
> > >>>>>>> easy. For now, one of the most blocker issues is the lack of
> > support
> > >> of
> > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not
> want
> > >> to
> > >>>>>> stay
> > >>>>>>> on the bridge api, we want to fully migrate to log4j2 finally, so
> > >>>>>>> supporting the above grammar in log4j1 bridge is not enough.
> > >>>>>>>
> > >>>>>>> This is the main log4j configuration file for hadoop
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> > >>>>>>>
> > >>>>>>> We have this in the file
> > >>>>>>>
> > >>>>>>> # Define the root logger to the system property
> > "hadoop.root.logger".
> > >>>>>>>> log4j.rootLogger=${hadoop.root.logger}
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> So our users could simply pass -Dhadoop.root.logger to control
> the
> > >>>> level
> > >>>>>>> and appender.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> > >>>>>>>
> > >>>>>>> Here we use an environment variable in our start up scripts. And
> I
> > >>>>>> believe
> > >>>>>>> there are lots of other hadoop deployment systems which will have
> > >> their
> > >>>>>> own
> > >>>>>>> start scripts.
> > >>>>>>>
> > >>>>>>> So in general, it is just impossible for us to drop the
> > >>>>>>> -Dhadoop.root.logger way of configuring our logging system.
> > >>>>>>>
> > >>>>>>> So here I want to ask if it is possible for log4j2 to still
> support
> > >> the
> > >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must
> > add
> > >> it
> > >>>>>>> back with no change, we just need a way to configure the level
> and
> > >>>>>> appeners
> > >>>>>>> at once, so something like
> > >>>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> > >>>>>>>
> > >>>>>>> Thanks.
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Volkan Yazıcı <vo...@yazi.ci>.
Giving it a one more try:
<CompoundProperty>
<Source>${sys:hadoop.root.logger}</Source>
<Split delimiter="\s+" trim="true" blanksExcluded="true">
<item index="0">hadoop.root.logger.level</item>
<slice startIndex="1">hadoop.root.logger.appender</slice>
</Split>
</CompoundProperty>
On Mon, Jan 10, 2022 at 3:54 PM Ralph Goers <ra...@dslextreme.com>
wrote:
> Yeah, that syntax is better although it doesn’t solve the multiple
> appender refs case either.
>
> LoggerConfig accepts a List<AppenderRef> so it would seem that if we
> modify LoggerConfig
> to add another attribute that is a string that creates an AppenderRef for
> each item then
> one could do
>
> <Logger name=“xyzzy” level=“INFO” appenderRefNames=“a, b, c”/>
>
>
> Ralph
>
> > On Jan 10, 2022, at 5:04 AM, Volkan Yazıcı <vo...@yazi.ci> wrote:
> >
> > I am curious if shall we adopt a more generic approach to extraction:
> >
> > <CompoundProperty>
> > <Source>${sys:hadoop.root.logger}</Source>
> > <Targets pattern="^(.+)\s*,\s(.+)$">
> > <Target>hadoop.root.logger.level</Target>
> > <Target>hadoop.root.logger.appender</Target>
> > </Targets>
> > </CompoundProperty>
> >
> > On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers <ra...@dslextreme.com>
> > wrote:
> >
> >> OK, I had a suspicion the reason for needing this was something like
> that.
> >>
> >> The syntax you propose below would require modifying PropertiesUtil, the
> >> Property
> >> plugin and StrSubstitutor to support arrays. I don’t think I’d want to
> go
> >> down that path.
> >> StrSubstitutor is already complicated and the idea of enhancing it to
> >> support stuff like
> >> this doesn’t excite me.
> >>
> >> I think we’d be better off creating a new plugin for this. Something
> like:
> >>
> >> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
> >> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
> >> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
> >> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
> >>
> >> The only problem here is if you want to provide more than one appender
> >> this won’t
> >> work well. But I am sure we could enhance AppenderRef to accept a list
> of
> >> appenders.
> >>
> >> Ralph
> >>
> >>
> >>
> >>
> >>> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> >>>
> >>> Thank you for the reply.
> >>>
> >>> And yes, it is not only for the root logger, we have tons of different
> >>> loggers in hadoop, as hadoop is constructed with several different
> >> projects
> >>> actually, and they are developed by different groups of people...
> >>>
> >>> And on splitting the config in shell, actually this is exactly what I
> >> have
> >>> done in HBase
> >>>
> >>> See
> >>>
> >>
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
> >>>
> >>> And also
> >>>
> >>
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
> >>>
> >>> For HBase maybe it is acceptable but hadoop is another story.
> >>> Hadoop is widely used almost in every bigdata platform. Among the
> >>> companies I've worked with, they all have their own hadoop deployment
> >>> systems which have modified version of start up scripts...
> >>>
> >>> I can imagine that if we do the same trick in HBase, then people will
> ask
> >>> again and again on the mailing list why passing -Dhadoop.root.logger
> does
> >>> not work, as well as other options like -Dyarn.root.logger,
> >>> -Dhadoop.security.logger, etc...
> >>>
> >>> So it will be good if we can support configure level and appenders in
> one
> >>> property.
> >>>
> >>> Or maybe another way is to enhance the ability in the Properties
> section,
> >>> so we can split one property into two properties, something like
> >>>
> >>> <Properties>
> >>> <Property name="hadoop.root.logger"
> >>> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
> >>> <Property
> >>> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
> >>> <Property name="hadoop.root.logger.appender"> ${hadoop.root.logger[1]}
> >>> </Property>
> >>> <Properties>
> >>>
> >>> Notice that since we could add multiple appenders here, maybe we need
> to
> >>> support something like ${hadoop.root.logger[1:] to combine all the
> >>> appenders...
> >>> This could also solve the problem as people could still pass
> >>> -Dhadoop.root.logger.
> >>>
> >>> But I'm not sure if adding the ability to split a string in
> configuation
> >>> could introduce new possible security concerns...
> >>>
> >>> Thanks.
> >>>
> >>> Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
> >>>
> >>>> We already support this with two variables. But if they only want to
> >> pass
> >>>> one then we have two options:
> >>>> 1. Modify PropertiesConfiguration to support a new element that
> allows a
> >>>> Level and AppenderRef which then gets split internally.
> >>>> 2. Create a Lookup that extracts the relevant portion of the data.
> >>>>
> >>>> Ralph
> >>>>
> >>>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <ga...@gmail.com>
> >> wrote:
> >>>>>
> >>>>> I think it is reasonable to say we can support this through 2 instead
> >> of
> >>>> 1
> >>>>> variable.
> >>>>>
> >>>>> Duo?
> >>>>>
> >>>>> Gary
> >>>>>
> >>>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <ra...@dslextreme.com>
> >>>> wrote:
> >>>>>
> >>>>>> I’m looking at this and have a couple of concerns. The script has
> >>>>>>
> >>>>>>
> >>>>>> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
> >>>>>>
> >>>>>>
> >>>>
> >>
> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
> >>>>>> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
> >>>>>>
> >>>>>> So it seems you need this for more than just the root logger.
> >>>>>>
> >>>>>> Second, you are asking us to accept “level, appender” as the value
> >> and
> >>>>>> under the covers split the
> >>>>>> value and assign the level to the level attribute and the appender
> >> name
> >>>> to
> >>>>>> the AppenderRef.
> >>>>>>
> >>>>>> While this certainly can be done it seems like it would be just as
> >> easy
> >>>> to
> >>>>>> do the split in the script
> >>>>>> and create two different variables for the logging configuration to
> >> pick
> >>>>>> up. Is there a reason that
> >>>>>> you can’t do this - for example perhaps users have hacked your
> scripts
> >>>> and
> >>>>>> now you can’t
> >>>>>> change them without breaking them? If so, was this “supported”?
> >>>>>>
> >>>>>> Ralph
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>> I brought this up in the incubator mailing list, and was suggested
> to
> >>>>>>> report directly to the log4j community.
> >>>>>>>
> >>>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
> >>>>>>>
> >>>>>>> In hadoop we started to try migrating to log4j2 long ago, but it is
> >> not
> >>>>>>> easy. For now, one of the most blocker issues is the lack of
> support
> >> of
> >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want
> >> to
> >>>>>> stay
> >>>>>>> on the bridge api, we want to fully migrate to log4j2 finally, so
> >>>>>>> supporting the above grammar in log4j1 bridge is not enough.
> >>>>>>>
> >>>>>>> This is the main log4j configuration file for hadoop
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> >>>>>>>
> >>>>>>> We have this in the file
> >>>>>>>
> >>>>>>> # Define the root logger to the system property
> "hadoop.root.logger".
> >>>>>>>> log4j.rootLogger=${hadoop.root.logger}
> >>>>>>>
> >>>>>>>
> >>>>>>> So our users could simply pass -Dhadoop.root.logger to control the
> >>>> level
> >>>>>>> and appender.
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> >>>>>>>
> >>>>>>> Here we use an environment variable in our start up scripts. And I
> >>>>>> believe
> >>>>>>> there are lots of other hadoop deployment systems which will have
> >> their
> >>>>>> own
> >>>>>>> start scripts.
> >>>>>>>
> >>>>>>> So in general, it is just impossible for us to drop the
> >>>>>>> -Dhadoop.root.logger way of configuring our logging system.
> >>>>>>>
> >>>>>>> So here I want to ask if it is possible for log4j2 to still support
> >> the
> >>>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must
> add
> >> it
> >>>>>>> back with no change, we just need a way to configure the level and
> >>>>>> appeners
> >>>>>>> at once, so something like
> >>>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Ralph Goers <ra...@dslextreme.com>.
Yeah, that syntax is better although it doesn’t solve the multiple appender refs case either.
LoggerConfig accepts a List<AppenderRef> so it would seem that if we modify LoggerConfig
to add another attribute that is a string that creates an AppenderRef for each item then
one could do
<Logger name=“xyzzy” level=“INFO” appenderRefNames=“a, b, c”/>
Ralph
> On Jan 10, 2022, at 5:04 AM, Volkan Yazıcı <vo...@yazi.ci> wrote:
>
> I am curious if shall we adopt a more generic approach to extraction:
>
> <CompoundProperty>
> <Source>${sys:hadoop.root.logger}</Source>
> <Targets pattern="^(.+)\s*,\s(.+)$">
> <Target>hadoop.root.logger.level</Target>
> <Target>hadoop.root.logger.appender</Target>
> </Targets>
> </CompoundProperty>
>
> On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers <ra...@dslextreme.com>
> wrote:
>
>> OK, I had a suspicion the reason for needing this was something like that.
>>
>> The syntax you propose below would require modifying PropertiesUtil, the
>> Property
>> plugin and StrSubstitutor to support arrays. I don’t think I’d want to go
>> down that path.
>> StrSubstitutor is already complicated and the idea of enhancing it to
>> support stuff like
>> this doesn’t excite me.
>>
>> I think we’d be better off creating a new plugin for this. Something like:
>>
>> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
>> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
>> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
>> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
>>
>> The only problem here is if you want to provide more than one appender
>> this won’t
>> work well. But I am sure we could enhance AppenderRef to accept a list of
>> appenders.
>>
>> Ralph
>>
>>
>>
>>
>>> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>>>
>>> Thank you for the reply.
>>>
>>> And yes, it is not only for the root logger, we have tons of different
>>> loggers in hadoop, as hadoop is constructed with several different
>> projects
>>> actually, and they are developed by different groups of people...
>>>
>>> And on splitting the config in shell, actually this is exactly what I
>> have
>>> done in HBase
>>>
>>> See
>>>
>> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
>>>
>>> And also
>>>
>> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
>>>
>>> For HBase maybe it is acceptable but hadoop is another story.
>>> Hadoop is widely used almost in every bigdata platform. Among the
>>> companies I've worked with, they all have their own hadoop deployment
>>> systems which have modified version of start up scripts...
>>>
>>> I can imagine that if we do the same trick in HBase, then people will ask
>>> again and again on the mailing list why passing -Dhadoop.root.logger does
>>> not work, as well as other options like -Dyarn.root.logger,
>>> -Dhadoop.security.logger, etc...
>>>
>>> So it will be good if we can support configure level and appenders in one
>>> property.
>>>
>>> Or maybe another way is to enhance the ability in the Properties section,
>>> so we can split one property into two properties, something like
>>>
>>> <Properties>
>>> <Property name="hadoop.root.logger"
>>> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
>>> <Property
>>> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
>>> <Property name="hadoop.root.logger.appender"> ${hadoop.root.logger[1]}
>>> </Property>
>>> <Properties>
>>>
>>> Notice that since we could add multiple appenders here, maybe we need to
>>> support something like ${hadoop.root.logger[1:] to combine all the
>>> appenders...
>>> This could also solve the problem as people could still pass
>>> -Dhadoop.root.logger.
>>>
>>> But I'm not sure if adding the ability to split a string in configuation
>>> could introduce new possible security concerns...
>>>
>>> Thanks.
>>>
>>> Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
>>>
>>>> We already support this with two variables. But if they only want to
>> pass
>>>> one then we have two options:
>>>> 1. Modify PropertiesConfiguration to support a new element that allows a
>>>> Level and AppenderRef which then gets split internally.
>>>> 2. Create a Lookup that extracts the relevant portion of the data.
>>>>
>>>> Ralph
>>>>
>>>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <ga...@gmail.com>
>> wrote:
>>>>>
>>>>> I think it is reasonable to say we can support this through 2 instead
>> of
>>>> 1
>>>>> variable.
>>>>>
>>>>> Duo?
>>>>>
>>>>> Gary
>>>>>
>>>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <ra...@dslextreme.com>
>>>> wrote:
>>>>>
>>>>>> I’m looking at this and have a couple of concerns. The script has
>>>>>>
>>>>>>
>>>>>> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
>>>>>>
>>>>>>
>>>>
>> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
>>>>>> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
>>>>>>
>>>>>> So it seems you need this for more than just the root logger.
>>>>>>
>>>>>> Second, you are asking us to accept “level, appender” as the value
>> and
>>>>>> under the covers split the
>>>>>> value and assign the level to the level attribute and the appender
>> name
>>>> to
>>>>>> the AppenderRef.
>>>>>>
>>>>>> While this certainly can be done it seems like it would be just as
>> easy
>>>> to
>>>>>> do the split in the script
>>>>>> and create two different variables for the logging configuration to
>> pick
>>>>>> up. Is there a reason that
>>>>>> you can’t do this - for example perhaps users have hacked your scripts
>>>> and
>>>>>> now you can’t
>>>>>> change them without breaking them? If so, was this “supported”?
>>>>>>
>>>>>> Ralph
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>> I brought this up in the incubator mailing list, and was suggested to
>>>>>>> report directly to the log4j community.
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
>>>>>>>
>>>>>>> In hadoop we started to try migrating to log4j2 long ago, but it is
>> not
>>>>>>> easy. For now, one of the most blocker issues is the lack of support
>> of
>>>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want
>> to
>>>>>> stay
>>>>>>> on the bridge api, we want to fully migrate to log4j2 finally, so
>>>>>>> supporting the above grammar in log4j1 bridge is not enough.
>>>>>>>
>>>>>>> This is the main log4j configuration file for hadoop
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
>>>>>>>
>>>>>>> We have this in the file
>>>>>>>
>>>>>>> # Define the root logger to the system property "hadoop.root.logger".
>>>>>>>> log4j.rootLogger=${hadoop.root.logger}
>>>>>>>
>>>>>>>
>>>>>>> So our users could simply pass -Dhadoop.root.logger to control the
>>>> level
>>>>>>> and appender.
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
>>>>>>>
>>>>>>> Here we use an environment variable in our start up scripts. And I
>>>>>> believe
>>>>>>> there are lots of other hadoop deployment systems which will have
>> their
>>>>>> own
>>>>>>> start scripts.
>>>>>>>
>>>>>>> So in general, it is just impossible for us to drop the
>>>>>>> -Dhadoop.root.logger way of configuring our logging system.
>>>>>>>
>>>>>>> So here I want to ask if it is possible for log4j2 to still support
>> the
>>>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add
>> it
>>>>>>> back with no change, we just need a way to configure the level and
>>>>>> appeners
>>>>>>> at once, so something like
>>>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
>>>>>>>
>>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Volkan Yazıcı <vo...@yazi.ci>.
I am curious if shall we adopt a more generic approach to extraction:
<CompoundProperty>
<Source>${sys:hadoop.root.logger}</Source>
<Targets pattern="^(.+)\s*,\s(.+)$">
<Target>hadoop.root.logger.level</Target>
<Target>hadoop.root.logger.appender</Target>
</Targets>
</CompoundProperty>
On Mon, Jan 10, 2022 at 5:53 AM Ralph Goers <ra...@dslextreme.com>
wrote:
> OK, I had a suspicion the reason for needing this was something like that.
>
> The syntax you propose below would require modifying PropertiesUtil, the
> Property
> plugin and StrSubstitutor to support arrays. I don’t think I’d want to go
> down that path.
> StrSubstitutor is already complicated and the idea of enhancing it to
> support stuff like
> this doesn’t excite me.
>
> I think we’d be better off creating a new plugin for this. Something like:
>
> <CompoundProperty name=“hadoop.root.logger.level” split=“,”
> index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
> <CompoundProperty name=“hadoop.root.logger.appender” split=“,”
> index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
>
> The only problem here is if you want to provide more than one appender
> this won’t
> work well. But I am sure we could enhance AppenderRef to accept a list of
> appenders.
>
> Ralph
>
>
>
>
> > On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> > Thank you for the reply.
> >
> > And yes, it is not only for the root logger, we have tons of different
> > loggers in hadoop, as hadoop is constructed with several different
> projects
> > actually, and they are developed by different groups of people...
> >
> > And on splitting the config in shell, actually this is exactly what I
> have
> > done in HBase
> >
> > See
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
> >
> > And also
> >
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
> >
> > For HBase maybe it is acceptable but hadoop is another story.
> > Hadoop is widely used almost in every bigdata platform. Among the
> > companies I've worked with, they all have their own hadoop deployment
> > systems which have modified version of start up scripts...
> >
> > I can imagine that if we do the same trick in HBase, then people will ask
> > again and again on the mailing list why passing -Dhadoop.root.logger does
> > not work, as well as other options like -Dyarn.root.logger,
> > -Dhadoop.security.logger, etc...
> >
> > So it will be good if we can support configure level and appenders in one
> > property.
> >
> > Or maybe another way is to enhance the ability in the Properties section,
> > so we can split one property into two properties, something like
> >
> > <Properties>
> > <Property name="hadoop.root.logger"
> > type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
> > <Property
> > name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
> > <Property name="hadoop.root.logger.appender"> ${hadoop.root.logger[1]}
> > </Property>
> > <Properties>
> >
> > Notice that since we could add multiple appenders here, maybe we need to
> > support something like ${hadoop.root.logger[1:] to combine all the
> > appenders...
> > This could also solve the problem as people could still pass
> > -Dhadoop.root.logger.
> >
> > But I'm not sure if adding the ability to split a string in configuation
> > could introduce new possible security concerns...
> >
> > Thanks.
> >
> > Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
> >
> >> We already support this with two variables. But if they only want to
> pass
> >> one then we have two options:
> >> 1. Modify PropertiesConfiguration to support a new element that allows a
> >> Level and AppenderRef which then gets split internally.
> >> 2. Create a Lookup that extracts the relevant portion of the data.
> >>
> >> Ralph
> >>
> >>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <ga...@gmail.com>
> wrote:
> >>>
> >>> I think it is reasonable to say we can support this through 2 instead
> of
> >> 1
> >>> variable.
> >>>
> >>> Duo?
> >>>
> >>> Gary
> >>>
> >>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <ra...@dslextreme.com>
> >> wrote:
> >>>
> >>>> I’m looking at this and have a couple of concerns. The script has
> >>>>
> >>>>
> >>>> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
> >>>>
> >>>>
> >>
> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
> >>>> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
> >>>>
> >>>> So it seems you need this for more than just the root logger.
> >>>>
> >>>> Second, you are asking us to accept “level, appender” as the value
> and
> >>>> under the covers split the
> >>>> value and assign the level to the level attribute and the appender
> name
> >> to
> >>>> the AppenderRef.
> >>>>
> >>>> While this certainly can be done it seems like it would be just as
> easy
> >> to
> >>>> do the split in the script
> >>>> and create two different variables for the logging configuration to
> pick
> >>>> up. Is there a reason that
> >>>> you can’t do this - for example perhaps users have hacked your scripts
> >> and
> >>>> now you can’t
> >>>> change them without breaking them? If so, was this “supported”?
> >>>>
> >>>> Ralph
> >>>>
> >>>>
> >>>>
> >>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> >> wrote:
> >>>>>
> >>>>> I brought this up in the incubator mailing list, and was suggested to
> >>>>> report directly to the log4j community.
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/HADOOP-16206
> >>>>>
> >>>>> In hadoop we started to try migrating to log4j2 long ago, but it is
> not
> >>>>> easy. For now, one of the most blocker issues is the lack of support
> of
> >>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want
> to
> >>>> stay
> >>>>> on the bridge api, we want to fully migrate to log4j2 finally, so
> >>>>> supporting the above grammar in log4j1 bridge is not enough.
> >>>>>
> >>>>> This is the main log4j configuration file for hadoop
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> >>>>>
> >>>>> We have this in the file
> >>>>>
> >>>>> # Define the root logger to the system property "hadoop.root.logger".
> >>>>>> log4j.rootLogger=${hadoop.root.logger}
> >>>>>
> >>>>>
> >>>>> So our users could simply pass -Dhadoop.root.logger to control the
> >> level
> >>>>> and appender.
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> >>>>>
> >>>>> Here we use an environment variable in our start up scripts. And I
> >>>> believe
> >>>>> there are lots of other hadoop deployment systems which will have
> their
> >>>> own
> >>>>> start scripts.
> >>>>>
> >>>>> So in general, it is just impossible for us to drop the
> >>>>> -Dhadoop.root.logger way of configuring our logging system.
> >>>>>
> >>>>> So here I want to ask if it is possible for log4j2 to still support
> the
> >>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add
> it
> >>>>> back with no change, we just need a way to configure the level and
> >>>> appeners
> >>>>> at once, so something like
> >>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> >>>>>
> >>>>> Thanks.
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Ralph Goers <ra...@dslextreme.com>.
OK, I had a suspicion the reason for needing this was something like that.
The syntax you propose below would require modifying PropertiesUtil, the Property
plugin and StrSubstitutor to support arrays. I don’t think I’d want to go down that path.
StrSubstitutor is already complicated and the idea of enhancing it to support stuff like
this doesn’t excite me.
I think we’d be better off creating a new plugin for this. Something like:
<CompoundProperty name=“hadoop.root.logger.level” split=“,” index=“0”>${sys:hadoop.root.logger}</CompoundProperty>
<CompoundProperty name=“hadoop.root.logger.appender” split=“,” index=“1”>${sys:hadoop.root.logger}</CompoundProperty>
The only problem here is if you want to provide more than one appender this won’t
work well. But I am sure we could enhance AppenderRef to accept a list of appenders.
Ralph
> On Jan 9, 2022, at 5:29 PM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> Thank you for the reply.
>
> And yes, it is not only for the root logger, we have tons of different
> loggers in hadoop, as hadoop is constructed with several different projects
> actually, and they are developed by different groups of people...
>
> And on splitting the config in shell, actually this is exactly what I have
> done in HBase
>
> See
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
>
> And also
> https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
>
> For HBase maybe it is acceptable but hadoop is another story.
> Hadoop is widely used almost in every bigdata platform. Among the
> companies I've worked with, they all have their own hadoop deployment
> systems which have modified version of start up scripts...
>
> I can imagine that if we do the same trick in HBase, then people will ask
> again and again on the mailing list why passing -Dhadoop.root.logger does
> not work, as well as other options like -Dyarn.root.logger,
> -Dhadoop.security.logger, etc...
>
> So it will be good if we can support configure level and appenders in one
> property.
>
> Or maybe another way is to enhance the ability in the Properties section,
> so we can split one property into two properties, something like
>
> <Properties>
> <Property name="hadoop.root.logger"
> type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
> <Property
> name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
> <Property name="hadoop.root.logger.appender"> ${hadoop.root.logger[1]}
> </Property>
> <Properties>
>
> Notice that since we could add multiple appenders here, maybe we need to
> support something like ${hadoop.root.logger[1:] to combine all the
> appenders...
> This could also solve the problem as people could still pass
> -Dhadoop.root.logger.
>
> But I'm not sure if adding the ability to split a string in configuation
> could introduce new possible security concerns...
>
> Thanks.
>
> Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
>
>> We already support this with two variables. But if they only want to pass
>> one then we have two options:
>> 1. Modify PropertiesConfiguration to support a new element that allows a
>> Level and AppenderRef which then gets split internally.
>> 2. Create a Lookup that extracts the relevant portion of the data.
>>
>> Ralph
>>
>>> On Jan 9, 2022, at 3:08 PM, Gary Gregory <ga...@gmail.com> wrote:
>>>
>>> I think it is reasonable to say we can support this through 2 instead of
>> 1
>>> variable.
>>>
>>> Duo?
>>>
>>> Gary
>>>
>>> On Sun, Jan 9, 2022, 16:24 Ralph Goers <ra...@dslextreme.com>
>> wrote:
>>>
>>>> I’m looking at this and have a couple of concerns. The script has
>>>>
>>>>
>>>> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
>>>>
>>>>
>> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
>>>> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
>>>>
>>>> So it seems you need this for more than just the root logger.
>>>>
>>>> Second, you are asking us to accept “level, appender” as the value and
>>>> under the covers split the
>>>> value and assign the level to the level attribute and the appender name
>> to
>>>> the AppenderRef.
>>>>
>>>> While this certainly can be done it seems like it would be just as easy
>> to
>>>> do the split in the script
>>>> and create two different variables for the logging configuration to pick
>>>> up. Is there a reason that
>>>> you can’t do this - for example perhaps users have hacked your scripts
>> and
>>>> now you can’t
>>>> change them without breaking them? If so, was this “supported”?
>>>>
>>>> Ralph
>>>>
>>>>
>>>>
>>>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>>>>>
>>>>> I brought this up in the incubator mailing list, and was suggested to
>>>>> report directly to the log4j community.
>>>>>
>>>>> https://issues.apache.org/jira/browse/HADOOP-16206
>>>>>
>>>>> In hadoop we started to try migrating to log4j2 long ago, but it is not
>>>>> easy. For now, one of the most blocker issues is the lack of support of
>>>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to
>>>> stay
>>>>> on the bridge api, we want to fully migrate to log4j2 finally, so
>>>>> supporting the above grammar in log4j1 bridge is not enough.
>>>>>
>>>>> This is the main log4j configuration file for hadoop
>>>>>
>>>>>
>>>>
>> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
>>>>>
>>>>> We have this in the file
>>>>>
>>>>> # Define the root logger to the system property "hadoop.root.logger".
>>>>>> log4j.rootLogger=${hadoop.root.logger}
>>>>>
>>>>>
>>>>> So our users could simply pass -Dhadoop.root.logger to control the
>> level
>>>>> and appender.
>>>>>
>>>>>
>>>>
>> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
>>>>>
>>>>> Here we use an environment variable in our start up scripts. And I
>>>> believe
>>>>> there are lots of other hadoop deployment systems which will have their
>>>> own
>>>>> start scripts.
>>>>>
>>>>> So in general, it is just impossible for us to drop the
>>>>> -Dhadoop.root.logger way of configuring our logging system.
>>>>>
>>>>> So here I want to ask if it is possible for log4j2 to still support the
>>>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
>>>>> back with no change, we just need a way to configure the level and
>>>> appeners
>>>>> at once, so something like
>>>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
>>>>>
>>>>> Thanks.
>>>>
>>>>
>>
>>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
Thank you for the reply.
And yes, it is not only for the root logger, we have tons of different
loggers in hadoop, as hadoop is constructed with several different projects
actually, and they are developed by different groups of people...
And on splitting the config in shell, actually this is exactly what I have
done in HBase
See
https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase#L829
And also
https://github.com/apache/hbase/blob/3ec3df5887e9271f7e75779eafe2439012cfb2c3/bin/hbase.cmd#L336
For HBase maybe it is acceptable but hadoop is another story.
Hadoop is widely used almost in every bigdata platform. Among the
companies I've worked with, they all have their own hadoop deployment
systems which have modified version of start up scripts...
I can imagine that if we do the same trick in HBase, then people will ask
again and again on the mailing list why passing -Dhadoop.root.logger does
not work, as well as other options like -Dyarn.root.logger,
-Dhadoop.security.logger, etc...
So it will be good if we can support configure level and appenders in one
property.
Or maybe another way is to enhance the ability in the Properties section,
so we can split one property into two properties, something like
<Properties>
<Property name="hadoop.root.logger"
type="array">${sys:hadoop.root.logger:-INFO,Console}</Property>
<Property
name="hadoop.root.logger.level">${hadoop.root.logger[0]}</Property>
<Property name="hadoop.root.logger.appender"> ${hadoop.root.logger[1]}
</Property>
<Properties>
Notice that since we could add multiple appenders here, maybe we need to
support something like ${hadoop.root.logger[1:] to combine all the
appenders...
This could also solve the problem as people could still pass
-Dhadoop.root.logger.
But I'm not sure if adding the ability to split a string in configuation
could introduce new possible security concerns...
Thanks.
Ralph Goers <ra...@dslextreme.com> 于2022年1月10日周一 07:14写道：
> We already support this with two variables. But if they only want to pass
> one then we have two options:
> 1. Modify PropertiesConfiguration to support a new element that allows a
> Level and AppenderRef which then gets split internally.
> 2. Create a Lookup that extracts the relevant portion of the data.
>
> Ralph
>
> > On Jan 9, 2022, at 3:08 PM, Gary Gregory <ga...@gmail.com> wrote:
> >
> > I think it is reasonable to say we can support this through 2 instead of
> 1
> > variable.
> >
> > Duo?
> >
> > Gary
> >
> > On Sun, Jan 9, 2022, 16:24 Ralph Goers <ra...@dslextreme.com>
> wrote:
> >
> >> I’m looking at this and have a couple of concerns. The script has
> >>
> >>
> >> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
> >>
> >>
> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
> >> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
> >>
> >> So it seems you need this for more than just the root logger.
> >>
> >> Second, you are asking us to accept “level, appender” as the value and
> >> under the covers split the
> >> value and assign the level to the level attribute and the appender name
> to
> >> the AppenderRef.
> >>
> >> While this certainly can be done it seems like it would be just as easy
> to
> >> do the split in the script
> >> and create two different variables for the logging configuration to pick
> >> up. Is there a reason that
> >> you can’t do this - for example perhaps users have hacked your scripts
> and
> >> now you can’t
> >> change them without breaking them? If so, was this “supported”?
> >>
> >> Ralph
> >>
> >>
> >>
> >>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> >>>
> >>> I brought this up in the incubator mailing list, and was suggested to
> >>> report directly to the log4j community.
> >>>
> >>> https://issues.apache.org/jira/browse/HADOOP-16206
> >>>
> >>> In hadoop we started to try migrating to log4j2 long ago, but it is not
> >>> easy. For now, one of the most blocker issues is the lack of support of
> >>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to
> >> stay
> >>> on the bridge api, we want to fully migrate to log4j2 finally, so
> >>> supporting the above grammar in log4j1 bridge is not enough.
> >>>
> >>> This is the main log4j configuration file for hadoop
> >>>
> >>>
> >>
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> >>>
> >>> We have this in the file
> >>>
> >>> # Define the root logger to the system property "hadoop.root.logger".
> >>>> log4j.rootLogger=${hadoop.root.logger}
> >>>
> >>>
> >>> So our users could simply pass -Dhadoop.root.logger to control the
> level
> >>> and appender.
> >>>
> >>>
> >>
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> >>>
> >>> Here we use an environment variable in our start up scripts. And I
> >> believe
> >>> there are lots of other hadoop deployment systems which will have their
> >> own
> >>> start scripts.
> >>>
> >>> So in general, it is just impossible for us to drop the
> >>> -Dhadoop.root.logger way of configuring our logging system.
> >>>
> >>> So here I want to ask if it is possible for log4j2 to still support the
> >>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
> >>> back with no change, we just need a way to configure the level and
> >> appeners
> >>> at once, so something like
> >>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> >>>
> >>> Thanks.
> >>
> >>
>
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Ralph Goers <ra...@dslextreme.com>.
We already support this with two variables. But if they only want to pass one then we have two options:
1. Modify PropertiesConfiguration to support a new element that allows a Level and AppenderRef which then gets split internally.
2. Create a Lookup that extracts the relevant portion of the data.
Ralph
> On Jan 9, 2022, at 3:08 PM, Gary Gregory <ga...@gmail.com> wrote:
>
> I think it is reasonable to say we can support this through 2 instead of 1
> variable.
>
> Duo?
>
> Gary
>
> On Sun, Jan 9, 2022, 16:24 Ralph Goers <ra...@dslextreme.com> wrote:
>
>> I’m looking at this and have a couple of concerns. The script has
>>
>>
>> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
>>
>> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
>> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
>>
>> So it seems you need this for more than just the root logger.
>>
>> Second, you are asking us to accept “level, appender” as the value and
>> under the covers split the
>> value and assign the level to the level attribute and the appender name to
>> the AppenderRef.
>>
>> While this certainly can be done it seems like it would be just as easy to
>> do the split in the script
>> and create two different variables for the logging configuration to pick
>> up. Is there a reason that
>> you can’t do this - for example perhaps users have hacked your scripts and
>> now you can’t
>> change them without breaking them? If so, was this “supported”?
>>
>> Ralph
>>
>>
>>
>>> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>>>
>>> I brought this up in the incubator mailing list, and was suggested to
>>> report directly to the log4j community.
>>>
>>> https://issues.apache.org/jira/browse/HADOOP-16206
>>>
>>> In hadoop we started to try migrating to log4j2 long ago, but it is not
>>> easy. For now, one of the most blocker issues is the lack of support of
>>> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to
>> stay
>>> on the bridge api, we want to fully migrate to log4j2 finally, so
>>> supporting the above grammar in log4j1 bridge is not enough.
>>>
>>> This is the main log4j configuration file for hadoop
>>>
>>>
>> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
>>>
>>> We have this in the file
>>>
>>> # Define the root logger to the system property "hadoop.root.logger".
>>>> log4j.rootLogger=${hadoop.root.logger}
>>>
>>>
>>> So our users could simply pass -Dhadoop.root.logger to control the level
>>> and appender.
>>>
>>>
>> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
>>>
>>> Here we use an environment variable in our start up scripts. And I
>> believe
>>> there are lots of other hadoop deployment systems which will have their
>> own
>>> start scripts.
>>>
>>> So in general, it is just impossible for us to drop the
>>> -Dhadoop.root.logger way of configuring our logging system.
>>>
>>> So here I want to ask if it is possible for log4j2 to still support the
>>> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
>>> back with no change, we just need a way to configure the level and
>> appeners
>>> at once, so something like
>>> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
>>>
>>> Thanks.
>>
>>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Gary Gregory <ga...@gmail.com>.
I think it is reasonable to say we can support this through 2 instead of 1
variable.
Duo?
Gary
On Sun, Jan 9, 2022, 16:24 Ralph Goers <ra...@dslextreme.com> wrote:
> I’m looking at this and have a couple of concerns. The script has
>
>
> HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
>
> HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
> HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
>
> So it seems you need this for more than just the root logger.
>
> Second, you are asking us to accept “level, appender” as the value and
> under the covers split the
> value and assign the level to the level attribute and the appender name to
> the AppenderRef.
>
> While this certainly can be done it seems like it would be just as easy to
> do the split in the script
> and create two different variables for the logging configuration to pick
> up. Is there a reason that
> you can’t do this - for example perhaps users have hacked your scripts and
> now you can’t
> change them without breaking them? If so, was this “supported”?
>
> Ralph
>
>
>
> > On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> >
> > I brought this up in the incubator mailing list, and was suggested to
> > report directly to the log4j community.
> >
> > https://issues.apache.org/jira/browse/HADOOP-16206
> >
> > In hadoop we started to try migrating to log4j2 long ago, but it is not
> > easy. For now, one of the most blocker issues is the lack of support of
> > 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to
> stay
> > on the bridge api, we want to fully migrate to log4j2 finally, so
> > supporting the above grammar in log4j1 bridge is not enough.
> >
> > This is the main log4j configuration file for hadoop
> >
> >
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
> >
> > We have this in the file
> >
> > # Define the root logger to the system property "hadoop.root.logger".
> >> log4j.rootLogger=${hadoop.root.logger}
> >
> >
> > So our users could simply pass -Dhadoop.root.logger to control the level
> > and appender.
> >
> >
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
> >
> > Here we use an environment variable in our start up scripts. And I
> believe
> > there are lots of other hadoop deployment systems which will have their
> own
> > start scripts.
> >
> > So in general, it is just impossible for us to drop the
> > -Dhadoop.root.logger way of configuring our logging system.
> >
> > So here I want to ask if it is possible for log4j2 to still support the
> > 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
> > back with no change, we just need a way to configure the level and
> appeners
> > at once, so something like
> > 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
> >
> > Thanks.
>
>
Re: [DISCUSS] Support 'log4j.rootLogger=INFO,Console' configuration grammar in log4j2
Posted by Ralph Goers <ra...@dslextreme.com>.
I’m looking at this and have a couple of concerns. The script has
HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-${HADOOP_LOGLEVEL},console}
HADOOP_DAEMON_ROOT_LOGGER=${HADOOP_DAEMON_ROOT_LOGGER:-${HADOOP_LOGLEVEL},RFA}
HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}
So it seems you need this for more than just the root logger.
Second, you are asking us to accept “level, appender” as the value and under the covers split the
value and assign the level to the level attribute and the appender name to the AppenderRef.
While this certainly can be done it seems like it would be just as easy to do the split in the script
and create two different variables for the logging configuration to pick up. Is there a reason that
you can’t do this - for example perhaps users have hacked your scripts and now you can’t
change them without breaking them? If so, was this “supported”?
Ralph
> On Jan 9, 2022, at 8:01 AM, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> I brought this up in the incubator mailing list, and was suggested to
> report directly to the log4j community.
>
> https://issues.apache.org/jira/browse/HADOOP-16206
>
> In hadoop we started to try migrating to log4j2 long ago, but it is not
> easy. For now, one of the most blocker issues is the lack of support of
> 'log4j.rootLogger=INFO,Console' grammar. Notice that we do not want to stay
> on the bridge api, we want to fully migrate to log4j2 finally, so
> supporting the above grammar in log4j1 bridge is not enough.
>
> This is the main log4j configuration file for hadoop
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
>
> We have this in the file
>
> # Define the root logger to the system property "hadoop.root.logger".
>> log4j.rootLogger=${hadoop.root.logger}
>
>
> So our users could simply pass -Dhadoop.root.logger to control the level
> and appender.
>
> https://github.com/apache/hadoop/blob/39efbc6b6fe53abed15f5639edcbaaa2a9dda6d2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L904
>
> Here we use an environment variable in our start up scripts. And I believe
> there are lots of other hadoop deployment systems which will have their own
> start scripts.
>
> So in general, it is just impossible for us to drop the
> -Dhadoop.root.logger way of configuring our logging system.
>
> So here I want to ask if it is possible for log4j2 to still support the
> 'log4j.rootLogger=INFO,Console' grammar. I'm not saying you must add it
> back with no change, we just need a way to configure the level and appeners
> at once, so something like
> 'log4j2.rootLogger.levelAndAppender=INFO,Console' is also OK.
>
> Thanks.