[DISCUSS] Log4j2 in ZooKeeper

[Tamas Penzes <ta...@cloudera.com.INVALID>]

Hi All,

I would open a discussion about log4j2 update.
Would we consider going up to log4j2 in a minor release (e.g. 3.7) or only
in a major one, like 4.0?
The latest log4j1 version (1.2.17) is really old and vulnerable, but log4j2
has a different config format, which means users should adopt their config
files when updating ZooKeeper.
Afaik we are compatible with both of them because of slf4j, but the default
is log4j1 at the moment.

What do you think about going up to log4j2 with 3.7?

Thanks, Tamaas

Re: [DISCUSS] Log4j2 in ZooKeeper

[Enrico Olivelli <eo...@gmail.com>]

Il Gio 8 Ott 2020, 20:24 Patrick Hunt <ph...@apache.org> ha scritto:

> On Thu, Oct 8, 2020 at 11:00 AM Tamas Penzes <ta...@cloudera.com.invalid>
> wrote:
>
> > Hi All,
> >
> > I would open a discussion about log4j2 update.
> > Would we consider going up to log4j2 in a minor release (e.g. 3.7) or
> only
> > in a major one, like 4.0?
> > The latest log4j1 version (1.2.17) is really old and vulnerable, but
> log4j2
> > has a different config format, which means users should adopt their
> config
> > files when updating ZooKeeper.
> > Afaik we are compatible with both of them because of slf4j, but the
> default
> > is log4j1 at the moment.
> >
> > What do you think about going up to log4j2 with 3.7?
>

I am fine with 3.7. As we are using slf4j users will be able to revert just
by trading the appropriate jars inside the lib directory.

>
> >
> Tamaas there's lots of background on this jira:
> https://issues.apache.org/jira/browse/ZOOKEEPER-2342
> In particular concern with b/w compat. There is also a patch attached.
>
> Is there a way we can provide run time selection without impacting code in
> a non-bw compatible way? Have other projects been able to solve this?
>

The main concern is about the logging configuration file on the server,
isn't it?
I don't know if it is common to override it.

On the client side it is slf4j that rules.

Enrico


> Patrick
>
>
> > Thanks, Tamaas
> >
>

Re: [DISCUSS] Log4j2 in ZooKeeper

[Enrico Olivelli <eo...@gmail.com>]

Il giorno mer 21 ott 2020 alle ore 04:14 Christopher <ct...@apache.org>
ha scritto:

> On Tue, Oct 20, 2020 at 7:43 PM Patrick Hunt <ph...@apache.org> wrote:
> >
> > On Tue, Oct 20, 2020 at 1:18 PM Tamas Penzes <tamaas@cloudera.com.invalid
> >
> > wrote:
> >
> > > Hi Patrick,
> > >
> > > I've read the ticket you have linked and also have used my time to ask
> > > after log4j2 backward compatibility and got quite negative feedback.
> > >
> > > It looks like we don't have a proper solution for it. We are (most
> > > probably) able to provide runtime selection of the logging framework,
> but
> > > we would still need a new log4j2 configuration file.
> > > There is a bw-compatibility layer of log4j configuration, but my
> sources
> > > told me not to use it as its functionality is very limited and it never
> > > became stable enough to use it in production.
> > >
> > > So the users must create a new log4j config file for log4j2, but I
> don't
> > > think it's a huge problem for ZooKeeper as it is small and its logging
> > > configuration isn't complex either.
> > > Doing it at the same time as updating ZooKeeper looks feasible and
> this is
> > > why I think we should not invest into the log4j version runtime
> selection.
> > >
> > > One day we have to do this step, my question is when?
> > > Should we wait for version 4 or can we do it in a "minor release", like
> > > 3.7. But ZK minor releases are like major releases on other Apache
> > > components.
> > >
> > >
> > What would a migration look like? Could we provide sufficient help for
> > people to easily migrate? It's not clear to me - if someone is using zk
> as
> > a client library and we upgrade log4j to log4j2 do they also need to
> > upgrade the code they are writing? (the code depending/using the zk
> jars).
>

In my opinion it is only a matter of configuration files and config scripts.
We can check for some good migration guides for the configuration files and
simply state clearly in the release notes that we did this switch and add a
link to the guide (an official guide from log4j people would be the best
choice).




> > I suspect embedding of zk server is also a possibility,


Yes it is a possibility, but as we are using slf4j usually you provide your
preferred slf4j imlpementation.
Enrico


> although less
> > common, what would the impact be there? I believe the answer should be
> only
> > the configs but perhaps we should experiment, say by doing a poc with
> hbase
> > (any project relying on zk) and then comparing the impact?
> >
>
> I migrated Accumulo to log4j2 back in February, if you want to see
> what that looked like:
> https://github.com/apache/accumulo/pull/1528/files
> (it's not as bad as it looks... we had a lot of log4j test config
> files, and some of that PR is related to a logging sink web service
> that wouldn't apply to ZK; it's mostly just changing config file
> formats and pom.xml dependency sections).
>
> If ZK client code (or an embedded server) is just using slf4j, then ZK
> does not need to care about what the logging implementation is...
> users can very easily use whatever they wish by simply ensuring their
> preferred implementation jars are on the client class path, along with
> any implementation-specific Java system properties or config files. By
> default, of course, the implementation jars and config files in the
> distribution tarball should be log4j2's jars and config file.
>
> The biggest hindrance is the internal use of log4j v1 APIs, such as
> the JMX stuff. In my opinion, that just needs to be removed. ZK should
> not depend on log4j to provide anything other than a runtime
> implementation of slf4j. I ran into an issue with that JMX stuff which
> I addressed in https://github.com/apache/zookeeper/pull/1270
>
> ZK's migration should be much less painful than Accumulo's, but I
> agree with Tamaas that it's not worth providing any special sauce in
> ZK to support runtime selection. Luckily, you don't have to... just
> use slf4j and users already get runtime selection for free. I also
> don't think it's a big burden to migrate to the new config file
> format(s) that are available with log4j2 when a user upgrades ZK.
>
> For what it's worth, If ZK is ready to make a clean break and drop all
> log4j v1 stuffs (including ripping out the JMX stuff that is holding
> it back from being pure slf4j), then I am willing to help with
> migrating config files and pom.xml dependencies.
>
> Christopher
>

Re: [DISCUSS] Log4j2 in ZooKeeper

[Christopher <ct...@apache.org>]

On Tue, Oct 20, 2020 at 7:43 PM Patrick Hunt <ph...@apache.org> wrote:
>
> On Tue, Oct 20, 2020 at 1:18 PM Tamas Penzes <ta...@cloudera.com.invalid>
> wrote:
>
> > Hi Patrick,
> >
> > I've read the ticket you have linked and also have used my time to ask
> > after log4j2 backward compatibility and got quite negative feedback.
> >
> > It looks like we don't have a proper solution for it. We are (most
> > probably) able to provide runtime selection of the logging framework, but
> > we would still need a new log4j2 configuration file.
> > There is a bw-compatibility layer of log4j configuration, but my sources
> > told me not to use it as its functionality is very limited and it never
> > became stable enough to use it in production.
> >
> > So the users must create a new log4j config file for log4j2, but I don't
> > think it's a huge problem for ZooKeeper as it is small and its logging
> > configuration isn't complex either.
> > Doing it at the same time as updating ZooKeeper looks feasible and this is
> > why I think we should not invest into the log4j version runtime selection.
> >
> > One day we have to do this step, my question is when?
> > Should we wait for version 4 or can we do it in a "minor release", like
> > 3.7. But ZK minor releases are like major releases on other Apache
> > components.
> >
> >
> What would a migration look like? Could we provide sufficient help for
> people to easily migrate? It's not clear to me - if someone is using zk as
> a client library and we upgrade log4j to log4j2 do they also need to
> upgrade the code they are writing? (the code depending/using the zk jars).
> I suspect embedding of zk server is also a possibility, although less
> common, what would the impact be there? I believe the answer should be only
> the configs but perhaps we should experiment, say by doing a poc with hbase
> (any project relying on zk) and then comparing the impact?
>

I migrated Accumulo to log4j2 back in February, if you want to see
what that looked like:
https://github.com/apache/accumulo/pull/1528/files
(it's not as bad as it looks... we had a lot of log4j test config
files, and some of that PR is related to a logging sink web service
that wouldn't apply to ZK; it's mostly just changing config file
formats and pom.xml dependency sections).

If ZK client code (or an embedded server) is just using slf4j, then ZK
does not need to care about what the logging implementation is...
users can very easily use whatever they wish by simply ensuring their
preferred implementation jars are on the client class path, along with
any implementation-specific Java system properties or config files. By
default, of course, the implementation jars and config files in the
distribution tarball should be log4j2's jars and config file.

The biggest hindrance is the internal use of log4j v1 APIs, such as
the JMX stuff. In my opinion, that just needs to be removed. ZK should
not depend on log4j to provide anything other than a runtime
implementation of slf4j. I ran into an issue with that JMX stuff which
I addressed in https://github.com/apache/zookeeper/pull/1270

ZK's migration should be much less painful than Accumulo's, but I
agree with Tamaas that it's not worth providing any special sauce in
ZK to support runtime selection. Luckily, you don't have to... just
use slf4j and users already get runtime selection for free. I also
don't think it's a big burden to migrate to the new config file
format(s) that are available with log4j2 when a user upgrades ZK.

For what it's worth, If ZK is ready to make a clean break and drop all
log4j v1 stuffs (including ripping out the JMX stuff that is holding
it back from being pure slf4j), then I am willing to help with
migrating config files and pom.xml dependencies.

Christopher

Re: [DISCUSS] Log4j2 in ZooKeeper

[Patrick Hunt <ph...@apache.org>]

On Tue, Oct 20, 2020 at 1:18 PM Tamas Penzes <ta...@cloudera.com.invalid>
wrote:

> Hi Patrick,
>
> I've read the ticket you have linked and also have used my time to ask
> after log4j2 backward compatibility and got quite negative feedback.
>
> It looks like we don't have a proper solution for it. We are (most
> probably) able to provide runtime selection of the logging framework, but
> we would still need a new log4j2 configuration file.
> There is a bw-compatibility layer of log4j configuration, but my sources
> told me not to use it as its functionality is very limited and it never
> became stable enough to use it in production.
>
> So the users must create a new log4j config file for log4j2, but I don't
> think it's a huge problem for ZooKeeper as it is small and its logging
> configuration isn't complex either.
> Doing it at the same time as updating ZooKeeper looks feasible and this is
> why I think we should not invest into the log4j version runtime selection.
>
> One day we have to do this step, my question is when?
> Should we wait for version 4 or can we do it in a "minor release", like
> 3.7. But ZK minor releases are like major releases on other Apache
> components.
>
>
What would a migration look like? Could we provide sufficient help for
people to easily migrate? It's not clear to me - if someone is using zk as
a client library and we upgrade log4j to log4j2 do they also need to
upgrade the code they are writing? (the code depending/using the zk jars).
I suspect embedding of zk server is also a possibility, although less
common, what would the impact be there? I believe the answer should be only
the configs but perhaps we should experiment, say by doing a poc with hbase
(any project relying on zk) and then comparing the impact?

Patrick


> Regards, Tamaas
>
>
> On Thu, Oct 8, 2020 at 8:24 PM Patrick Hunt <ph...@apache.org> wrote:
>
> > On Thu, Oct 8, 2020 at 11:00 AM Tamas Penzes <tamaas@cloudera.com.invalid
> >
> > wrote:
> >
> > > Hi All,
> > >
> > > I would open a discussion about log4j2 update.
> > > Would we consider going up to log4j2 in a minor release (e.g. 3.7) or
> > only
> > > in a major one, like 4.0?
> > > The latest log4j1 version (1.2.17) is really old and vulnerable, but
> > log4j2
> > > has a different config format, which means users should adopt their
> > config
> > > files when updating ZooKeeper.
> > > Afaik we are compatible with both of them because of slf4j, but the
> > default
> > > is log4j1 at the moment.
> > >
> > > What do you think about going up to log4j2 with 3.7?
> > >
> > >
> > Tamaas there's lots of background on this jira:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-2342
> > In particular concern with b/w compat. There is also a patch attached.
> >
> > Is there a way we can provide run time selection without impacting code
> in
> > a non-bw compatible way? Have other projects been able to solve this?
> >
> > Patrick
> >
> >
> > > Thanks, Tamaas
> > >
> >
>

Re: [DISCUSS] Log4j2 in ZooKeeper

[Tamas Penzes <ta...@cloudera.com.INVALID>]

Hi Patrick,

I've read the ticket you have linked and also have used my time to ask
after log4j2 backward compatibility and got quite negative feedback.

It looks like we don't have a proper solution for it. We are (most
probably) able to provide runtime selection of the logging framework, but
we would still need a new log4j2 configuration file.
There is a bw-compatibility layer of log4j configuration, but my sources
told me not to use it as its functionality is very limited and it never
became stable enough to use it in production.

So the users must create a new log4j config file for log4j2, but I don't
think it's a huge problem for ZooKeeper as it is small and its logging
configuration isn't complex either.
Doing it at the same time as updating ZooKeeper looks feasible and this is
why I think we should not invest into the log4j version runtime selection.

One day we have to do this step, my question is when?
Should we wait for version 4 or can we do it in a "minor release", like
3.7. But ZK minor releases are like major releases on other Apache
components.

Regards, Tamaas


On Thu, Oct 8, 2020 at 8:24 PM Patrick Hunt <ph...@apache.org> wrote:

> On Thu, Oct 8, 2020 at 11:00 AM Tamas Penzes <ta...@cloudera.com.invalid>
> wrote:
>
> > Hi All,
> >
> > I would open a discussion about log4j2 update.
> > Would we consider going up to log4j2 in a minor release (e.g. 3.7) or
> only
> > in a major one, like 4.0?
> > The latest log4j1 version (1.2.17) is really old and vulnerable, but
> log4j2
> > has a different config format, which means users should adopt their
> config
> > files when updating ZooKeeper.
> > Afaik we are compatible with both of them because of slf4j, but the
> default
> > is log4j1 at the moment.
> >
> > What do you think about going up to log4j2 with 3.7?
> >
> >
> Tamaas there's lots of background on this jira:
> https://issues.apache.org/jira/browse/ZOOKEEPER-2342
> In particular concern with b/w compat. There is also a patch attached.
>
> Is there a way we can provide run time selection without impacting code in
> a non-bw compatible way? Have other projects been able to solve this?
>
> Patrick
>
>
> > Thanks, Tamaas
> >
>

Re: [DISCUSS] Log4j2 in ZooKeeper

[Patrick Hunt <ph...@apache.org>]

On Thu, Oct 8, 2020 at 11:00 AM Tamas Penzes <ta...@cloudera.com.invalid>
wrote:

> Hi All,
>
> I would open a discussion about log4j2 update.
> Would we consider going up to log4j2 in a minor release (e.g. 3.7) or only
> in a major one, like 4.0?
> The latest log4j1 version (1.2.17) is really old and vulnerable, but log4j2
> has a different config format, which means users should adopt their config
> files when updating ZooKeeper.
> Afaik we are compatible with both of them because of slf4j, but the default
> is log4j1 at the moment.
>
> What do you think about going up to log4j2 with 3.7?
>
>
Tamaas there's lots of background on this jira:
https://issues.apache.org/jira/browse/ZOOKEEPER-2342
In particular concern with b/w compat. There is also a patch attached.

Is there a way we can provide run time selection without impacting code in
a non-bw compatible way? Have other projects been able to solve this?

Patrick


> Thanks, Tamaas
>