You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Duo Zhang <zh...@apache.org> on 2022/01/20 13:55:42 UTC
[DISCUSS] Migrate hadoop from log4j1 to log4j2
There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
is time to speed up the migration to log4j2 work[4] now.
You can see the discussion on the jira issue[4], our goal is to fully
migrate to log4j2 and the current most blocking issue is lack of the
"log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
started a discussion thread on the log4j dev mailing list[5] and the result
is optimistic and I've filed an issue for log4j2[6], but I do not think it
could be addressed and released soon. If we want to fully migrate to
log4j2, then either we introduce new environment variables or split the old
HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
complexity of our current startup scripts, the work is not easy and it will
also break lots of other hadoop deployment systems if they do not use our
startup scripts...
So after reconsidering the current situation, I prefer we use the log4j1.2
bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
addressed and released, we start to fully migrate to log4j2. Of course we
have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
ContainerLogAppender and ContainerRollingLogAppender which inherit
FileAppender and RollingFileAppender in log4j1, which are not part of the
log4j1.2 bridge. But anyway, at least we could just copy the source code to
hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
do not have related CVEs.
Thoughts? For me I would like us to make a new 3.4.x release line to remove
the log4j1 dependencies ASAP.
Thanks.
1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
4. https://issues.apache.org/jira/browse/HADOOP-16206
5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
6. https://issues.apache.org/jira/browse/LOG4J2-3341
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Thu, 20 Jan 2022 at 17:15, Andrew Purtell <an...@gmail.com>
wrote:
> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.
i like this
>
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Thu, 20 Jan 2022 at 17:15, Andrew Purtell <an...@gmail.com>
wrote:
> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.
i like this
>
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Thu, 20 Jan 2022 at 17:15, Andrew Purtell <an...@gmail.com>
wrote:
> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.
i like this
>
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Wei-Chiu Chuang <we...@apache.org>.
+1 I think it makes sense to use reload4j in maint releases.
I have a draft PR doing this (https://github.com/apache/hadoop/pull/3906)
log4j2 in Hadoop 3.4.0 makes sense to me. There could be incompatibilities
introduced by log4j2, but I feel we should at least make it 3.4.0 a
"preview" release, and try to address the incompat in later versions (e.g.
3.4.1)
On Fri, Jan 21, 2022 at 8:42 AM Duo Zhang <zh...@apache.org> wrote:
> For maintenance release line I also support we switch to reload4j to
> address the security issues first. We could file an issue for it.
>
> Andrew Purtell <an...@gmail.com>于2022年1月21日 周五01:15写道：
>
> > Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> > LogBack) as a strategy for new releases, but you have the option in
> > maintenance releases to use Reload4J to maintain Appender API and
> > operational compatibility, and users who want to minimize risks in
> > production while mitigating the security issues will prefer that.
> >
> > > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > Reload4J has fixed all of those CVEs without requiring an upgrade.
> > >
> > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> > >>
> > >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I
> think
> > it
> > >> is time to speed up the migration to log4j2 work[4] now.
> > >>
> > >> You can see the discussion on the jira issue[4], our goal is to fully
> > >> migrate to log4j2 and the current most blocking issue is lack of the
> > >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've
> already
> > >> started a discussion thread on the log4j dev mailing list[5] and the
> > result
> > >> is optimistic and I've filed an issue for log4j2[6], but I do not
> think
> > it
> > >> could be addressed and released soon. If we want to fully migrate to
> > >> log4j2, then either we introduce new environment variables or split
> the
> > old
> > >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering
> the
> > >> complexity of our current startup scripts, the work is not easy and it
> > will
> > >> also break lots of other hadoop deployment systems if they do not use
> > our
> > >> startup scripts...
> > >>
> > >> So after reconsidering the current situation, I prefer we use the
> > log4j1.2
> > >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> > >> addressed and released, we start to fully migrate to log4j2. Of course
> > we
> > >> have other problems for log4j1.2 bridge too, as we have
> TaskLogAppender,
> > >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> > >> FileAppender and RollingFileAppender in log4j1, which are not part of
> > the
> > >> log4j1.2 bridge. But anyway, at least we could just copy the source
> > code to
> > >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> > classes
> > >> do not have related CVEs.
> > >>
> > >> Thoughts? For me I would like us to make a new 3.4.x release line to
> > remove
> > >> the log4j1 dependencies ASAP.
> > >>
> > >> Thanks.
> > >>
> > >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> > >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> > >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> > >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> > >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> > >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
> >
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Wei-Chiu Chuang <we...@apache.org>.
+1 I think it makes sense to use reload4j in maint releases.
I have a draft PR doing this (https://github.com/apache/hadoop/pull/3906)
log4j2 in Hadoop 3.4.0 makes sense to me. There could be incompatibilities
introduced by log4j2, but I feel we should at least make it 3.4.0 a
"preview" release, and try to address the incompat in later versions (e.g.
3.4.1)
On Fri, Jan 21, 2022 at 8:42 AM Duo Zhang <zh...@apache.org> wrote:
> For maintenance release line I also support we switch to reload4j to
> address the security issues first. We could file an issue for it.
>
> Andrew Purtell <an...@gmail.com>于2022年1月21日 周五01:15写道：
>
> > Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> > LogBack) as a strategy for new releases, but you have the option in
> > maintenance releases to use Reload4J to maintain Appender API and
> > operational compatibility, and users who want to minimize risks in
> > production while mitigating the security issues will prefer that.
> >
> > > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > Reload4J has fixed all of those CVEs without requiring an upgrade.
> > >
> > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> > >>
> > >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I
> think
> > it
> > >> is time to speed up the migration to log4j2 work[4] now.
> > >>
> > >> You can see the discussion on the jira issue[4], our goal is to fully
> > >> migrate to log4j2 and the current most blocking issue is lack of the
> > >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've
> already
> > >> started a discussion thread on the log4j dev mailing list[5] and the
> > result
> > >> is optimistic and I've filed an issue for log4j2[6], but I do not
> think
> > it
> > >> could be addressed and released soon. If we want to fully migrate to
> > >> log4j2, then either we introduce new environment variables or split
> the
> > old
> > >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering
> the
> > >> complexity of our current startup scripts, the work is not easy and it
> > will
> > >> also break lots of other hadoop deployment systems if they do not use
> > our
> > >> startup scripts...
> > >>
> > >> So after reconsidering the current situation, I prefer we use the
> > log4j1.2
> > >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> > >> addressed and released, we start to fully migrate to log4j2. Of course
> > we
> > >> have other problems for log4j1.2 bridge too, as we have
> TaskLogAppender,
> > >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> > >> FileAppender and RollingFileAppender in log4j1, which are not part of
> > the
> > >> log4j1.2 bridge. But anyway, at least we could just copy the source
> > code to
> > >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> > classes
> > >> do not have related CVEs.
> > >>
> > >> Thoughts? For me I would like us to make a new 3.4.x release line to
> > remove
> > >> the log4j1 dependencies ASAP.
> > >>
> > >> Thanks.
> > >>
> > >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> > >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> > >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> > >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> > >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> > >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
> >
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Wei-Chiu Chuang <we...@apache.org>.
+1 I think it makes sense to use reload4j in maint releases.
I have a draft PR doing this (https://github.com/apache/hadoop/pull/3906)
log4j2 in Hadoop 3.4.0 makes sense to me. There could be incompatibilities
introduced by log4j2, but I feel we should at least make it 3.4.0 a
"preview" release, and try to address the incompat in later versions (e.g.
3.4.1)
On Fri, Jan 21, 2022 at 8:42 AM Duo Zhang <zh...@apache.org> wrote:
> For maintenance release line I also support we switch to reload4j to
> address the security issues first. We could file an issue for it.
>
> Andrew Purtell <an...@gmail.com>于2022年1月21日 周五01:15写道：
>
> > Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> > LogBack) as a strategy for new releases, but you have the option in
> > maintenance releases to use Reload4J to maintain Appender API and
> > operational compatibility, and users who want to minimize risks in
> > production while mitigating the security issues will prefer that.
> >
> > > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > Reload4J has fixed all of those CVEs without requiring an upgrade.
> > >
> > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> > >>
> > >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I
> think
> > it
> > >> is time to speed up the migration to log4j2 work[4] now.
> > >>
> > >> You can see the discussion on the jira issue[4], our goal is to fully
> > >> migrate to log4j2 and the current most blocking issue is lack of the
> > >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've
> already
> > >> started a discussion thread on the log4j dev mailing list[5] and the
> > result
> > >> is optimistic and I've filed an issue for log4j2[6], but I do not
> think
> > it
> > >> could be addressed and released soon. If we want to fully migrate to
> > >> log4j2, then either we introduce new environment variables or split
> the
> > old
> > >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering
> the
> > >> complexity of our current startup scripts, the work is not easy and it
> > will
> > >> also break lots of other hadoop deployment systems if they do not use
> > our
> > >> startup scripts...
> > >>
> > >> So after reconsidering the current situation, I prefer we use the
> > log4j1.2
> > >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> > >> addressed and released, we start to fully migrate to log4j2. Of course
> > we
> > >> have other problems for log4j1.2 bridge too, as we have
> TaskLogAppender,
> > >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> > >> FileAppender and RollingFileAppender in log4j1, which are not part of
> > the
> > >> log4j1.2 bridge. But anyway, at least we could just copy the source
> > code to
> > >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> > classes
> > >> do not have related CVEs.
> > >>
> > >> Thoughts? For me I would like us to make a new 3.4.x release line to
> > remove
> > >> the log4j1 dependencies ASAP.
> > >>
> > >> Thanks.
> > >>
> > >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> > >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> > >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> > >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> > >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> > >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
> >
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Wei-Chiu Chuang <we...@apache.org>.
+1 I think it makes sense to use reload4j in maint releases.
I have a draft PR doing this (https://github.com/apache/hadoop/pull/3906)
log4j2 in Hadoop 3.4.0 makes sense to me. There could be incompatibilities
introduced by log4j2, but I feel we should at least make it 3.4.0 a
"preview" release, and try to address the incompat in later versions (e.g.
3.4.1)
On Fri, Jan 21, 2022 at 8:42 AM Duo Zhang <zh...@apache.org> wrote:
> For maintenance release line I also support we switch to reload4j to
> address the security issues first. We could file an issue for it.
>
> Andrew Purtell <an...@gmail.com>于2022年1月21日 周五01:15写道：
>
> > Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> > LogBack) as a strategy for new releases, but you have the option in
> > maintenance releases to use Reload4J to maintain Appender API and
> > operational compatibility, and users who want to minimize risks in
> > production while mitigating the security issues will prefer that.
> >
> > > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > Reload4J has fixed all of those CVEs without requiring an upgrade.
> > >
> > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> > >>
> > >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I
> think
> > it
> > >> is time to speed up the migration to log4j2 work[4] now.
> > >>
> > >> You can see the discussion on the jira issue[4], our goal is to fully
> > >> migrate to log4j2 and the current most blocking issue is lack of the
> > >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've
> already
> > >> started a discussion thread on the log4j dev mailing list[5] and the
> > result
> > >> is optimistic and I've filed an issue for log4j2[6], but I do not
> think
> > it
> > >> could be addressed and released soon. If we want to fully migrate to
> > >> log4j2, then either we introduce new environment variables or split
> the
> > old
> > >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering
> the
> > >> complexity of our current startup scripts, the work is not easy and it
> > will
> > >> also break lots of other hadoop deployment systems if they do not use
> > our
> > >> startup scripts...
> > >>
> > >> So after reconsidering the current situation, I prefer we use the
> > log4j1.2
> > >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> > >> addressed and released, we start to fully migrate to log4j2. Of course
> > we
> > >> have other problems for log4j1.2 bridge too, as we have
> TaskLogAppender,
> > >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> > >> FileAppender and RollingFileAppender in log4j1, which are not part of
> > the
> > >> log4j1.2 bridge. But anyway, at least we could just copy the source
> > code to
> > >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> > classes
> > >> do not have related CVEs.
> > >>
> > >> Thoughts? For me I would like us to make a new 3.4.x release line to
> > remove
> > >> the log4j1 dependencies ASAP.
> > >>
> > >> Thanks.
> > >>
> > >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> > >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> > >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> > >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> > >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> > >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
> >
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Duo Zhang <zh...@apache.org>.
For maintenance release line I also support we switch to reload4j to
address the security issues first. We could file an issue for it.
Andrew Purtell <an...@gmail.com>于2022年1月21日 周五01:15写道：
> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Thu, 20 Jan 2022 at 17:15, Andrew Purtell <an...@gmail.com>
wrote:
> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.
i like this
>
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Duo Zhang <zh...@apache.org>.
For maintenance release line I also support we switch to reload4j to
address the security issues first. We could file an issue for it.
Andrew Purtell <an...@gmail.com>于2022年1月21日 周五01:15写道：
> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Duo Zhang <zh...@apache.org>.
For maintenance release line I also support we switch to reload4j to
address the security issues first. We could file an issue for it.
Andrew Purtell <an...@gmail.com>于2022年1月21日 周五01:15写道：
> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Duo Zhang <zh...@apache.org>.
For maintenance release line I also support we switch to reload4j to
address the security issues first. We could file an issue for it.
Andrew Purtell <an...@gmail.com>于2022年1月21日 周五01:15写道：
> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Andrew Purtell <an...@gmail.com>.
Just to clarify: I think you want to upgrade to Log4J2 (or switch to LogBack) as a strategy for new releases, but you have the option in maintenance releases to use Reload4J to maintain Appender API and operational compatibility, and users who want to minimize risks in production while mitigating the security issues will prefer that.
> On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com> wrote:
>
> Reload4J has fixed all of those CVEs without requiring an upgrade.
>
>> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
>>
>> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
>> is time to speed up the migration to log4j2 work[4] now.
>>
>> You can see the discussion on the jira issue[4], our goal is to fully
>> migrate to log4j2 and the current most blocking issue is lack of the
>> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
>> started a discussion thread on the log4j dev mailing list[5] and the result
>> is optimistic and I've filed an issue for log4j2[6], but I do not think it
>> could be addressed and released soon. If we want to fully migrate to
>> log4j2, then either we introduce new environment variables or split the old
>> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
>> complexity of our current startup scripts, the work is not easy and it will
>> also break lots of other hadoop deployment systems if they do not use our
>> startup scripts...
>>
>> So after reconsidering the current situation, I prefer we use the log4j1.2
>> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
>> addressed and released, we start to fully migrate to log4j2. Of course we
>> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
>> ContainerLogAppender and ContainerRollingLogAppender which inherit
>> FileAppender and RollingFileAppender in log4j1, which are not part of the
>> log4j1.2 bridge. But anyway, at least we could just copy the source code to
>> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
>> do not have related CVEs.
>>
>> Thoughts? For me I would like us to make a new 3.4.x release line to remove
>> the log4j1 dependencies ASAP.
>>
>> Thanks.
>>
>> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
>> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
>> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
>> 4. https://issues.apache.org/jira/browse/HADOOP-16206
>> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
>> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Andrew Purtell <an...@gmail.com>.
Just to clarify: I think you want to upgrade to Log4J2 (or switch to LogBack) as a strategy for new releases, but you have the option in maintenance releases to use Reload4J to maintain Appender API and operational compatibility, and users who want to minimize risks in production while mitigating the security issues will prefer that.
> On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com> wrote:
>
> Reload4J has fixed all of those CVEs without requiring an upgrade.
>
>> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
>>
>> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
>> is time to speed up the migration to log4j2 work[4] now.
>>
>> You can see the discussion on the jira issue[4], our goal is to fully
>> migrate to log4j2 and the current most blocking issue is lack of the
>> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
>> started a discussion thread on the log4j dev mailing list[5] and the result
>> is optimistic and I've filed an issue for log4j2[6], but I do not think it
>> could be addressed and released soon. If we want to fully migrate to
>> log4j2, then either we introduce new environment variables or split the old
>> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
>> complexity of our current startup scripts, the work is not easy and it will
>> also break lots of other hadoop deployment systems if they do not use our
>> startup scripts...
>>
>> So after reconsidering the current situation, I prefer we use the log4j1.2
>> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
>> addressed and released, we start to fully migrate to log4j2. Of course we
>> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
>> ContainerLogAppender and ContainerRollingLogAppender which inherit
>> FileAppender and RollingFileAppender in log4j1, which are not part of the
>> log4j1.2 bridge. But anyway, at least we could just copy the source code to
>> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
>> do not have related CVEs.
>>
>> Thoughts? For me I would like us to make a new 3.4.x release line to remove
>> the log4j1 dependencies ASAP.
>>
>> Thanks.
>>
>> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
>> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
>> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
>> 4. https://issues.apache.org/jira/browse/HADOOP-16206
>> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
>> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Andrew Purtell <an...@gmail.com>.
Just to clarify: I think you want to upgrade to Log4J2 (or switch to LogBack) as a strategy for new releases, but you have the option in maintenance releases to use Reload4J to maintain Appender API and operational compatibility, and users who want to minimize risks in production while mitigating the security issues will prefer that.
> On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com> wrote:
>
> Reload4J has fixed all of those CVEs without requiring an upgrade.
>
>> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
>>
>> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
>> is time to speed up the migration to log4j2 work[4] now.
>>
>> You can see the discussion on the jira issue[4], our goal is to fully
>> migrate to log4j2 and the current most blocking issue is lack of the
>> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
>> started a discussion thread on the log4j dev mailing list[5] and the result
>> is optimistic and I've filed an issue for log4j2[6], but I do not think it
>> could be addressed and released soon. If we want to fully migrate to
>> log4j2, then either we introduce new environment variables or split the old
>> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
>> complexity of our current startup scripts, the work is not easy and it will
>> also break lots of other hadoop deployment systems if they do not use our
>> startup scripts...
>>
>> So after reconsidering the current situation, I prefer we use the log4j1.2
>> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
>> addressed and released, we start to fully migrate to log4j2. Of course we
>> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
>> ContainerLogAppender and ContainerRollingLogAppender which inherit
>> FileAppender and RollingFileAppender in log4j1, which are not part of the
>> log4j1.2 bridge. But anyway, at least we could just copy the source code to
>> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
>> do not have related CVEs.
>>
>> Thoughts? For me I would like us to make a new 3.4.x release line to remove
>> the log4j1 dependencies ASAP.
>>
>> Thanks.
>>
>> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
>> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
>> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
>> 4. https://issues.apache.org/jira/browse/HADOOP-16206
>> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
>> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Andrew Purtell <an...@gmail.com>.
Just to clarify: I think you want to upgrade to Log4J2 (or switch to LogBack) as a strategy for new releases, but you have the option in maintenance releases to use Reload4J to maintain Appender API and operational compatibility, and users who want to minimize risks in production while mitigating the security issues will prefer that.
> On Jan 20, 2022, at 8:59 AM, Andrew Purtell <an...@gmail.com> wrote:
>
> Reload4J has fixed all of those CVEs without requiring an upgrade.
>
>> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
>>
>> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
>> is time to speed up the migration to log4j2 work[4] now.
>>
>> You can see the discussion on the jira issue[4], our goal is to fully
>> migrate to log4j2 and the current most blocking issue is lack of the
>> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
>> started a discussion thread on the log4j dev mailing list[5] and the result
>> is optimistic and I've filed an issue for log4j2[6], but I do not think it
>> could be addressed and released soon. If we want to fully migrate to
>> log4j2, then either we introduce new environment variables or split the old
>> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
>> complexity of our current startup scripts, the work is not easy and it will
>> also break lots of other hadoop deployment systems if they do not use our
>> startup scripts...
>>
>> So after reconsidering the current situation, I prefer we use the log4j1.2
>> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
>> addressed and released, we start to fully migrate to log4j2. Of course we
>> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
>> ContainerLogAppender and ContainerRollingLogAppender which inherit
>> FileAppender and RollingFileAppender in log4j1, which are not part of the
>> log4j1.2 bridge. But anyway, at least we could just copy the source code to
>> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
>> do not have related CVEs.
>>
>> Thoughts? For me I would like us to make a new 3.4.x release line to remove
>> the log4j1 dependencies ASAP.
>>
>> Thanks.
>>
>> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
>> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
>> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
>> 4. https://issues.apache.org/jira/browse/HADOOP-16206
>> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
>> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Andrew Purtell <an...@gmail.com>.
Reload4J has fixed all of those CVEs without requiring an upgrade.
> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
>
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
>
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
>
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
>
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
>
> Thanks.
>
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Duo Zhang <zh...@apache.org>.
The EventCounter class has already been removed in HADOOP-17524.
And on the filters, by default log4j1.2 bridge has log4j1 filter support,
but as said above, maybe it is not fully functional if you have some
advanced usage. So mind providing more information about how we use filters
in Hadoop?
Thanks.
Arpit Agarwal <aa...@cloudera.com.invalid>于2022年1月21日 周五00:44写道：
> Hi Duo,
>
> Thank you for starting this discussion. Log4j1.2 bridge seems like a
> practical short-term solution. However the bridge will silently affect
> applications that add appenders or filters. NameNode audit logger and
> metrics come to mind. There may be others.
>
> Thanks,
> Arpit
>
>
> > On Jan 20, 2022, at 5:55 AM, Duo Zhang <zh...@apache.org> wrote:
> >
> > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> > is time to speed up the migration to log4j2 work[4] now.
> >
> > You can see the discussion on the jira issue[4], our goal is to fully
> > migrate to log4j2 and the current most blocking issue is lack of the
> > "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> > started a discussion thread on the log4j dev mailing list[5] and the
> result
> > is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> > could be addressed and released soon. If we want to fully migrate to
> > log4j2, then either we introduce new environment variables or split the
> old
> > HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> > complexity of our current startup scripts, the work is not easy and it
> will
> > also break lots of other hadoop deployment systems if they do not use our
> > startup scripts...
> >
> > So after reconsidering the current situation, I prefer we use the
> log4j1.2
> > bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> > addressed and released, we start to fully migrate to log4j2. Of course we
> > have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> > ContainerLogAppender and ContainerRollingLogAppender which inherit
> > FileAppender and RollingFileAppender in log4j1, which are not part of the
> > log4j1.2 bridge. But anyway, at least we could just copy the source code
> to
> > hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> > do not have related CVEs.
> >
> > Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> > the log4j1 dependencies ASAP.
> >
> > Thanks.
> >
> > 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> > 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> > 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> > 4. https://issues.apache.org/jira/browse/HADOOP-16206
> > 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> > 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Duo Zhang <zh...@apache.org>.
The EventCounter class has already been removed in HADOOP-17524.
And on the filters, by default log4j1.2 bridge has log4j1 filter support,
but as said above, maybe it is not fully functional if you have some
advanced usage. So mind providing more information about how we use filters
in Hadoop?
Thanks.
Arpit Agarwal <aa...@cloudera.com.invalid>于2022年1月21日 周五00:44写道：
> Hi Duo,
>
> Thank you for starting this discussion. Log4j1.2 bridge seems like a
> practical short-term solution. However the bridge will silently affect
> applications that add appenders or filters. NameNode audit logger and
> metrics come to mind. There may be others.
>
> Thanks,
> Arpit
>
>
> > On Jan 20, 2022, at 5:55 AM, Duo Zhang <zh...@apache.org> wrote:
> >
> > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> > is time to speed up the migration to log4j2 work[4] now.
> >
> > You can see the discussion on the jira issue[4], our goal is to fully
> > migrate to log4j2 and the current most blocking issue is lack of the
> > "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> > started a discussion thread on the log4j dev mailing list[5] and the
> result
> > is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> > could be addressed and released soon. If we want to fully migrate to
> > log4j2, then either we introduce new environment variables or split the
> old
> > HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> > complexity of our current startup scripts, the work is not easy and it
> will
> > also break lots of other hadoop deployment systems if they do not use our
> > startup scripts...
> >
> > So after reconsidering the current situation, I prefer we use the
> log4j1.2
> > bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> > addressed and released, we start to fully migrate to log4j2. Of course we
> > have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> > ContainerLogAppender and ContainerRollingLogAppender which inherit
> > FileAppender and RollingFileAppender in log4j1, which are not part of the
> > log4j1.2 bridge. But anyway, at least we could just copy the source code
> to
> > hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> > do not have related CVEs.
> >
> > Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> > the log4j1 dependencies ASAP.
> >
> > Thanks.
> >
> > 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> > 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> > 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> > 4. https://issues.apache.org/jira/browse/HADOOP-16206
> > 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> > 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Duo Zhang <zh...@apache.org>.
The EventCounter class has already been removed in HADOOP-17524.
And on the filters, by default log4j1.2 bridge has log4j1 filter support,
but as said above, maybe it is not fully functional if you have some
advanced usage. So mind providing more information about how we use filters
in Hadoop?
Thanks.
Arpit Agarwal <aa...@cloudera.com.invalid>于2022年1月21日 周五00:44写道：
> Hi Duo,
>
> Thank you for starting this discussion. Log4j1.2 bridge seems like a
> practical short-term solution. However the bridge will silently affect
> applications that add appenders or filters. NameNode audit logger and
> metrics come to mind. There may be others.
>
> Thanks,
> Arpit
>
>
> > On Jan 20, 2022, at 5:55 AM, Duo Zhang <zh...@apache.org> wrote:
> >
> > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> > is time to speed up the migration to log4j2 work[4] now.
> >
> > You can see the discussion on the jira issue[4], our goal is to fully
> > migrate to log4j2 and the current most blocking issue is lack of the
> > "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> > started a discussion thread on the log4j dev mailing list[5] and the
> result
> > is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> > could be addressed and released soon. If we want to fully migrate to
> > log4j2, then either we introduce new environment variables or split the
> old
> > HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> > complexity of our current startup scripts, the work is not easy and it
> will
> > also break lots of other hadoop deployment systems if they do not use our
> > startup scripts...
> >
> > So after reconsidering the current situation, I prefer we use the
> log4j1.2
> > bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> > addressed and released, we start to fully migrate to log4j2. Of course we
> > have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> > ContainerLogAppender and ContainerRollingLogAppender which inherit
> > FileAppender and RollingFileAppender in log4j1, which are not part of the
> > log4j1.2 bridge. But anyway, at least we could just copy the source code
> to
> > hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> > do not have related CVEs.
> >
> > Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> > the log4j1 dependencies ASAP.
> >
> > Thanks.
> >
> > 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> > 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> > 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> > 4. https://issues.apache.org/jira/browse/HADOOP-16206
> > 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> > 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Duo Zhang <zh...@apache.org>.
The EventCounter class has already been removed in HADOOP-17524.
And on the filters, by default log4j1.2 bridge has log4j1 filter support,
but as said above, maybe it is not fully functional if you have some
advanced usage. So mind providing more information about how we use filters
in Hadoop?
Thanks.
Arpit Agarwal <aa...@cloudera.com.invalid>于2022年1月21日 周五00:44写道：
> Hi Duo,
>
> Thank you for starting this discussion. Log4j1.2 bridge seems like a
> practical short-term solution. However the bridge will silently affect
> applications that add appenders or filters. NameNode audit logger and
> metrics come to mind. There may be others.
>
> Thanks,
> Arpit
>
>
> > On Jan 20, 2022, at 5:55 AM, Duo Zhang <zh...@apache.org> wrote:
> >
> > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> > is time to speed up the migration to log4j2 work[4] now.
> >
> > You can see the discussion on the jira issue[4], our goal is to fully
> > migrate to log4j2 and the current most blocking issue is lack of the
> > "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> > started a discussion thread on the log4j dev mailing list[5] and the
> result
> > is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> > could be addressed and released soon. If we want to fully migrate to
> > log4j2, then either we introduce new environment variables or split the
> old
> > HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> > complexity of our current startup scripts, the work is not easy and it
> will
> > also break lots of other hadoop deployment systems if they do not use our
> > startup scripts...
> >
> > So after reconsidering the current situation, I prefer we use the
> log4j1.2
> > bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> > addressed and released, we start to fully migrate to log4j2. Of course we
> > have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> > ContainerLogAppender and ContainerRollingLogAppender which inherit
> > FileAppender and RollingFileAppender in log4j1, which are not part of the
> > log4j1.2 bridge. But anyway, at least we could just copy the source code
> to
> > hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> > do not have related CVEs.
> >
> > Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> > the log4j1 dependencies ASAP.
> >
> > Thanks.
> >
> > 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> > 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> > 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> > 4. https://issues.apache.org/jira/browse/HADOOP-16206
> > 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> > 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Arpit Agarwal <aa...@cloudera.com.INVALID>.
Hi Duo,
Thank you for starting this discussion. Log4j1.2 bridge seems like a practical short-term solution. However the bridge will silently affect applications that add appenders or filters. NameNode audit logger and metrics come to mind. There may be others.
Thanks,
Arpit
> On Jan 20, 2022, at 5:55 AM, Duo Zhang <zh...@apache.org> wrote:
>
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
>
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
>
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
>
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
>
> Thanks.
>
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Arpit Agarwal <aa...@cloudera.com.INVALID>.
Hi Duo,
Thank you for starting this discussion. Log4j1.2 bridge seems like a practical short-term solution. However the bridge will silently affect applications that add appenders or filters. NameNode audit logger and metrics come to mind. There may be others.
Thanks,
Arpit
> On Jan 20, 2022, at 5:55 AM, Duo Zhang <zh...@apache.org> wrote:
>
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
>
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
>
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
>
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
>
> Thanks.
>
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Andrew Purtell <an...@gmail.com>.
Reload4J has fixed all of those CVEs without requiring an upgrade.
> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
>
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
>
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
>
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
>
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
>
> Thanks.
>
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Andrew Purtell <an...@gmail.com>.
Reload4J has fixed all of those CVEs without requiring an upgrade.
> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
>
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
>
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
>
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
>
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
>
> Thanks.
>
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Arpit Agarwal <aa...@cloudera.com.INVALID>.
Hi Duo,
Thank you for starting this discussion. Log4j1.2 bridge seems like a practical short-term solution. However the bridge will silently affect applications that add appenders or filters. NameNode audit logger and metrics come to mind. There may be others.
Thanks,
Arpit
> On Jan 20, 2022, at 5:55 AM, Duo Zhang <zh...@apache.org> wrote:
>
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
>
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
>
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
>
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
>
> Thanks.
>
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Arpit Agarwal <aa...@cloudera.com.INVALID>.
Hi Duo,
Thank you for starting this discussion. Log4j1.2 bridge seems like a practical short-term solution. However the bridge will silently affect applications that add appenders or filters. NameNode audit logger and metrics come to mind. There may be others.
Thanks,
Arpit
> On Jan 20, 2022, at 5:55 AM, Duo Zhang <zh...@apache.org> wrote:
>
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
>
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
>
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
>
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
>
> Thanks.
>
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2
Posted by Andrew Purtell <an...@gmail.com>.
Reload4J has fixed all of those CVEs without requiring an upgrade.
> On Jan 20, 2022, at 5:56 AM, Duo Zhang <zh...@apache.org> wrote:
>
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
>
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
>
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
>
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
>
> Thanks.
>
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org