You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Mickael Maison <mi...@gmail.com> on 2024/11/06 09:47:17 UTC
Re: Configuration format for Log4j Core 2.x
Hi,
I think our example log4j2 file should be as idiomatic as possible. If
YAML is now the recommended format, then it makes sense to adopt it. I
would also do this directly in 4.0.0.
Like David, one concern when adding extra dependencies is CVEs. YAML
is a widely used format and the libraries are actively maintained so I
think it's acceptable.
In my opinion, the configuration format for Kafka is a completely
orthogonal issue. If people want to adopt YAML this can be done in a
separate discussion. I don't see issues with using YAML for
configuring log4j2 and properties for Kafka.
Thanks,
Mickael
On Thu, Oct 31, 2024 at 5:47 PM Piotr P. Karwasz
<pi...@mailing.copernik.eu> wrote:
>
> Hi Viktor,
>
> On 31.10.2024 10:19, Viktor Somogyi-Vass wrote:
> > I could see a transition for Kafka configs too to YAML. It's widely used in
> > other projects as well and for Kafka too it would make sense to group them
> > for instance by log, storage, networking, security etc.. It definitely
> > has an advantage that we could move to a more well defined structure and
> > away from these long prefixes which are very cumbersome in some places.
> > Perhaps 4.0 would have been a good candidate but given that we should
> > provide backward compatibility anyway, later is good too.
>
> To migrate the configuration from Java Properties to YAML and maintain
> BC, you can use `jackson-dataformat-properties`. This definitively can
> be done in a minor release.
>
> [1]
> https://github.com/FasterXML/jackson-dataformats-text/blob/master/properties/README.md
>
> > 4. Data bindings and parsers are common sources of CVEs. It looks like
> > Snakeyaml is no exception (
> > https://www.cvedetails.com/version-list/0/66013/1/), though it doesn't look
> > much worse than Jackson. Just to point out, this will add a bit of
> > dependency overhead as we keep up with security patches.
> >
> > It's a good point about CVEs. I haven't seen it on the dev list but were
> > there any conversations about enabling dependabot version updates? With
> > automatic dependabot PRs we could get fixes in as soon as it opens the PR.
>
> I don't know how it works in Gradle, but with Maven we don't get
> Dependabot PRs for transitive dependencies such as SnakeYAML. Since the
> dependency is not present in the `pom.xml`, its version can not be
> updated. Dependabot might be able to create alerts in the "Security" tab
> though.
>
> Since Log4j is a library without an executable distribution, it is not
> guaranteed that a new version of Log4j will be released each time a
> vulnerability in SnakeYAML is discovered and that vulnerability is
> exploitable in Log4j Core. It does not make much sense to make a new
> release just to upgrade a number in `pom.xml`.
>
> Until VEX-es can be created automatically, the only way I can think of
> to help the Kafka team with CVEs in transitive dependencies is a manual
> process:
>
> * You mark `jackson-dataformat-yaml` with a comment like "Used by Log4j
> Core".
>
> * If a CVE is reported against that artifact or its dependencies, you
> open an issue in Log4j[2], so we can advise on the exploitability of the
> CVE.
>
> * With that information you can decide whether to release a new Kafka
> version or not.
>
> Piotr
>
> [2] https://github.com/apache/logging-log4j2/issues
>
>
Re: Configuration format for Log4j Core 2.x
Posted by TengYao Chi <ki...@gmail.com>.
Hello Chia-Ping
I have just included the configuration migration section in the KIP-653
<https://cwiki.apache.org/confluence/x/GCx4CQ>.
Please take a look.
Best,
TengYao
Jörn Franke <jo...@gmail.com> 於 2024年11月19日 週二 上午12:36寫道：
> Yaml allows to add comments compared to JSON which is not only for
> examples good, but also to document certain implications when deploying
> production configuration.
>
> > Am 18.11.2024 um 15:02 schrieb TengYao Chi <ki...@gmail.com>:
> >
> > Hi everyone,
> >
> > From my experience, it has been quite challenging to understand and
> > transform the .properties file from log4j 1 to log4j 2, as the properties
> > format is fairly flat and difficult to understand.
> > In comparison, YAML is more structured, easier to read, and widely used.
> I
> > would like to +1 for using YAML properties instead.
> > If a KIP is required, I am willing to prepare one.
> >
> > Best,
> > TengYao
> >
> > Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午9:35寫道：
> >
> >> Hi Chia-Ping,
> >>
> >> Yes in 4.0.0 we will only provide log4j2 configuration files. We will
> >> still support log4j.properties files for compatibility but that will
> >> print a warning when these are used.
> >>
> >> However my question is about the format of our log4j2 files. Currently
> >> in the PR we are using the "properties" format (log4j2.properties).
> >> But as pointed by Piotr (from the Apache Logging PMC), log4j2 has a
> >> hierarchical structure which does not translate well into the
> >> properties format, so configuration file in this format have many
> >> quirks. Instead he suggested using XML, JSON or YAML in our example
> >> files (so log4j2.xml or log4j2.json or log4j2.yaml) as these are the
> >> preferred formats now.
> >>
> >> David, Viktor and myself have expressed our preference toward adopting
> >> YAML for our log4j2 configuration files. I'm wondering if that's
> >> enough consensus, or if people want to discuss this further, or even
> >> want a vote?
> >>
> >> Thanks,
> >> Mickael
> >>
> >>
> >>> On Mon, Nov 18, 2024 at 1:52 PM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >>>
> >>> hi, Mickael
> >>>
> >>> I'm +1 for using the log4j2 format. Otherwise, it's odd for users to
> see
> >>> deprecation warnings about log4j.properties when using our example
> files.
> >>>
> >>> Best,
> >>> Chia-Ping
> >>>
> >>>
> >>>
> >>> Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午7:33寫道：
> >>>
> >>>> Hi,
> >>>>
> >>>> The log4j2 migration PR is pretty much ready to be merged and the
> >>>> first deadlines for 4.0.0 is approaching fast.
> >>>> I think we should decide which format to use in our example log4j2
> >>>> files to avoid having to update the format shortly after 4.0.0.
> >>>>
> >>>> This point is not really covered by KIP-653: Upgrade log4j to log4j2.
> >>>> Do we want another KIP and vote or is a consensus emerging?
> >>>>
> >>>> Thanks,
> >>>> Mickael
> >>>>
> >>>> On Wed, Nov 6, 2024 at 10:47 AM Mickael Maison <
> >> mickael.maison@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I think our example log4j2 file should be as idiomatic as possible.
> >> If
> >>>>> YAML is now the recommended format, then it makes sense to adopt it.
> >> I
> >>>>> would also do this directly in 4.0.0.
> >>>>>
> >>>>> Like David, one concern when adding extra dependencies is CVEs. YAML
> >>>>> is a widely used format and the libraries are actively maintained so
> >> I
> >>>>> think it's acceptable.
> >>>>>
> >>>>> In my opinion, the configuration format for Kafka is a completely
> >>>>> orthogonal issue. If people want to adopt YAML this can be done in a
> >>>>> separate discussion. I don't see issues with using YAML for
> >>>>> configuring log4j2 and properties for Kafka.
> >>>>>
> >>>>> Thanks,
> >>>>> Mickael
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Oct 31, 2024 at 5:47 PM Piotr P. Karwasz
> >>>>> <pi...@mailing.copernik.eu> wrote:
> >>>>>>
> >>>>>> Hi Viktor,
> >>>>>>
> >>>>>> On 31.10.2024 10:19, Viktor Somogyi-Vass wrote:
> >>>>>>> I could see a transition for Kafka configs too to YAML. It's
> >> widely
> >>>> used in
> >>>>>>> other projects as well and for Kafka too it would make sense to
> >>>> group them
> >>>>>>> for instance by log, storage, networking, security etc.. It
> >>>> definitely
> >>>>>>> has an advantage that we could move to a more well defined
> >> structure
> >>>> and
> >>>>>>> away from these long prefixes which are very cumbersome in some
> >>>> places.
> >>>>>>> Perhaps 4.0 would have been a good candidate but given that we
> >> should
> >>>>>>> provide backward compatibility anyway, later is good too.
> >>>>>>
> >>>>>> To migrate the configuration from Java Properties to YAML and
> >> maintain
> >>>>>> BC, you can use `jackson-dataformat-properties`. This definitively
> >> can
> >>>>>> be done in a minor release.
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>
> >>
> https://github.com/FasterXML/jackson-dataformats-text/blob/master/properties/README.md
> >>>>>>
> >>>>>>> 4. Data bindings and parsers are common sources of CVEs. It looks
> >>>> like
> >>>>>>> Snakeyaml is no exception (
> >>>>>>> https://www.cvedetails.com/version-list/0/66013/1/), though it
> >>>> doesn't look
> >>>>>>> much worse than Jackson. Just to point out, this will add a bit
> >> of
> >>>>>>> dependency overhead as we keep up with security patches.
> >>>>>>>
> >>>>>>> It's a good point about CVEs. I haven't seen it on the dev list
> >> but
> >>>> were
> >>>>>>> there any conversations about enabling dependabot version
> >> updates?
> >>>> With
> >>>>>>> automatic dependabot PRs we could get fixes in as soon as it
> >> opens
> >>>> the PR.
> >>>>>>
> >>>>>> I don't know how it works in Gradle, but with Maven we don't get
> >>>>>> Dependabot PRs for transitive dependencies such as SnakeYAML.
> >> Since the
> >>>>>> dependency is not present in the `pom.xml`, its version can not be
> >>>>>> updated. Dependabot might be able to create alerts in the
> >> "Security"
> >>>> tab
> >>>>>> though.
> >>>>>>
> >>>>>> Since Log4j is a library without an executable distribution, it is
> >> not
> >>>>>> guaranteed that a new version of Log4j will be released each time a
> >>>>>> vulnerability in SnakeYAML is discovered and that vulnerability is
> >>>>>> exploitable in Log4j Core. It does not make much sense to make a
> >> new
> >>>>>> release just to upgrade a number in `pom.xml`.
> >>>>>>
> >>>>>> Until VEX-es can be created automatically, the only way I can
> >> think of
> >>>>>> to help the Kafka team with CVEs in transitive dependencies is a
> >> manual
> >>>>>> process:
> >>>>>>
> >>>>>> * You mark `jackson-dataformat-yaml` with a comment like "Used by
> >> Log4j
> >>>>>> Core".
> >>>>>>
> >>>>>> * If a CVE is reported against that artifact or its dependencies,
> >> you
> >>>>>> open an issue in Log4j[2], so we can advise on the exploitability
> >> of
> >>>> the
> >>>>>> CVE.
> >>>>>>
> >>>>>> * With that information you can decide whether to release a new
> >> Kafka
> >>>>>> version or not.
> >>>>>>
> >>>>>> Piotr
> >>>>>>
> >>>>>> [2] https://github.com/apache/logging-log4j2/issues
> >>>>>>
> >>>>>>
> >>>>
> >>
>
Re: Configuration format for Log4j Core 2.x
Posted by Jörn Franke <jo...@gmail.com>.
Yaml allows to add comments compared to JSON which is not only for examples good, but also to document certain implications when deploying production configuration.
> Am 18.11.2024 um 15:02 schrieb TengYao Chi <ki...@gmail.com>:
>
> Hi everyone,
>
> From my experience, it has been quite challenging to understand and
> transform the .properties file from log4j 1 to log4j 2, as the properties
> format is fairly flat and difficult to understand.
> In comparison, YAML is more structured, easier to read, and widely used. I
> would like to +1 for using YAML properties instead.
> If a KIP is required, I am willing to prepare one.
>
> Best,
> TengYao
>
> Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午9:35寫道：
>
>> Hi Chia-Ping,
>>
>> Yes in 4.0.0 we will only provide log4j2 configuration files. We will
>> still support log4j.properties files for compatibility but that will
>> print a warning when these are used.
>>
>> However my question is about the format of our log4j2 files. Currently
>> in the PR we are using the "properties" format (log4j2.properties).
>> But as pointed by Piotr (from the Apache Logging PMC), log4j2 has a
>> hierarchical structure which does not translate well into the
>> properties format, so configuration file in this format have many
>> quirks. Instead he suggested using XML, JSON or YAML in our example
>> files (so log4j2.xml or log4j2.json or log4j2.yaml) as these are the
>> preferred formats now.
>>
>> David, Viktor and myself have expressed our preference toward adopting
>> YAML for our log4j2 configuration files. I'm wondering if that's
>> enough consensus, or if people want to discuss this further, or even
>> want a vote?
>>
>> Thanks,
>> Mickael
>>
>>
>>> On Mon, Nov 18, 2024 at 1:52 PM Chia-Ping Tsai <ch...@gmail.com> wrote:
>>>
>>> hi, Mickael
>>>
>>> I'm +1 for using the log4j2 format. Otherwise, it's odd for users to see
>>> deprecation warnings about log4j.properties when using our example files.
>>>
>>> Best,
>>> Chia-Ping
>>>
>>>
>>>
>>> Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午7:33寫道：
>>>
>>>> Hi,
>>>>
>>>> The log4j2 migration PR is pretty much ready to be merged and the
>>>> first deadlines for 4.0.0 is approaching fast.
>>>> I think we should decide which format to use in our example log4j2
>>>> files to avoid having to update the format shortly after 4.0.0.
>>>>
>>>> This point is not really covered by KIP-653: Upgrade log4j to log4j2.
>>>> Do we want another KIP and vote or is a consensus emerging?
>>>>
>>>> Thanks,
>>>> Mickael
>>>>
>>>> On Wed, Nov 6, 2024 at 10:47 AM Mickael Maison <
>> mickael.maison@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I think our example log4j2 file should be as idiomatic as possible.
>> If
>>>>> YAML is now the recommended format, then it makes sense to adopt it.
>> I
>>>>> would also do this directly in 4.0.0.
>>>>>
>>>>> Like David, one concern when adding extra dependencies is CVEs. YAML
>>>>> is a widely used format and the libraries are actively maintained so
>> I
>>>>> think it's acceptable.
>>>>>
>>>>> In my opinion, the configuration format for Kafka is a completely
>>>>> orthogonal issue. If people want to adopt YAML this can be done in a
>>>>> separate discussion. I don't see issues with using YAML for
>>>>> configuring log4j2 and properties for Kafka.
>>>>>
>>>>> Thanks,
>>>>> Mickael
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 31, 2024 at 5:47 PM Piotr P. Karwasz
>>>>> <pi...@mailing.copernik.eu> wrote:
>>>>>>
>>>>>> Hi Viktor,
>>>>>>
>>>>>> On 31.10.2024 10:19, Viktor Somogyi-Vass wrote:
>>>>>>> I could see a transition for Kafka configs too to YAML. It's
>> widely
>>>> used in
>>>>>>> other projects as well and for Kafka too it would make sense to
>>>> group them
>>>>>>> for instance by log, storage, networking, security etc.. It
>>>> definitely
>>>>>>> has an advantage that we could move to a more well defined
>> structure
>>>> and
>>>>>>> away from these long prefixes which are very cumbersome in some
>>>> places.
>>>>>>> Perhaps 4.0 would have been a good candidate but given that we
>> should
>>>>>>> provide backward compatibility anyway, later is good too.
>>>>>>
>>>>>> To migrate the configuration from Java Properties to YAML and
>> maintain
>>>>>> BC, you can use `jackson-dataformat-properties`. This definitively
>> can
>>>>>> be done in a minor release.
>>>>>>
>>>>>> [1]
>>>>>>
>>>>
>> https://github.com/FasterXML/jackson-dataformats-text/blob/master/properties/README.md
>>>>>>
>>>>>>> 4. Data bindings and parsers are common sources of CVEs. It looks
>>>> like
>>>>>>> Snakeyaml is no exception (
>>>>>>> https://www.cvedetails.com/version-list/0/66013/1/), though it
>>>> doesn't look
>>>>>>> much worse than Jackson. Just to point out, this will add a bit
>> of
>>>>>>> dependency overhead as we keep up with security patches.
>>>>>>>
>>>>>>> It's a good point about CVEs. I haven't seen it on the dev list
>> but
>>>> were
>>>>>>> there any conversations about enabling dependabot version
>> updates?
>>>> With
>>>>>>> automatic dependabot PRs we could get fixes in as soon as it
>> opens
>>>> the PR.
>>>>>>
>>>>>> I don't know how it works in Gradle, but with Maven we don't get
>>>>>> Dependabot PRs for transitive dependencies such as SnakeYAML.
>> Since the
>>>>>> dependency is not present in the `pom.xml`, its version can not be
>>>>>> updated. Dependabot might be able to create alerts in the
>> "Security"
>>>> tab
>>>>>> though.
>>>>>>
>>>>>> Since Log4j is a library without an executable distribution, it is
>> not
>>>>>> guaranteed that a new version of Log4j will be released each time a
>>>>>> vulnerability in SnakeYAML is discovered and that vulnerability is
>>>>>> exploitable in Log4j Core. It does not make much sense to make a
>> new
>>>>>> release just to upgrade a number in `pom.xml`.
>>>>>>
>>>>>> Until VEX-es can be created automatically, the only way I can
>> think of
>>>>>> to help the Kafka team with CVEs in transitive dependencies is a
>> manual
>>>>>> process:
>>>>>>
>>>>>> * You mark `jackson-dataformat-yaml` with a comment like "Used by
>> Log4j
>>>>>> Core".
>>>>>>
>>>>>> * If a CVE is reported against that artifact or its dependencies,
>> you
>>>>>> open an issue in Log4j[2], so we can advise on the exploitability
>> of
>>>> the
>>>>>> CVE.
>>>>>>
>>>>>> * With that information you can decide whether to release a new
>> Kafka
>>>>>> version or not.
>>>>>>
>>>>>> Piotr
>>>>>>
>>>>>> [2] https://github.com/apache/logging-log4j2/issues
>>>>>>
>>>>>>
>>>>
>>
Re: Configuration format for Log4j Core 2.x
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Mickael,
Thanks for the clarification. I'm still +1 on using YAML, as this PR
ultimately changes the 'content' of the log4j properties file. So why not
use a more readable structure with this change? Additionally, users can
still use the deprecated properties when migrating to 4.0.
hi TengYao
Could you please update the KIP to add this change?
Best,
Chia-Ping
TengYao Chi <ki...@gmail.com> 於 2024年11月18日 週一 下午10:02寫道：
> Hi everyone,
>
> From my experience, it has been quite challenging to understand and
> transform the .properties file from log4j 1 to log4j 2, as the properties
> format is fairly flat and difficult to understand.
> In comparison, YAML is more structured, easier to read, and widely used. I
> would like to +1 for using YAML properties instead.
> If a KIP is required, I am willing to prepare one.
>
> Best,
> TengYao
>
> Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午9:35寫道：
>
> > Hi Chia-Ping,
> >
> > Yes in 4.0.0 we will only provide log4j2 configuration files. We will
> > still support log4j.properties files for compatibility but that will
> > print a warning when these are used.
> >
> > However my question is about the format of our log4j2 files. Currently
> > in the PR we are using the "properties" format (log4j2.properties).
> > But as pointed by Piotr (from the Apache Logging PMC), log4j2 has a
> > hierarchical structure which does not translate well into the
> > properties format, so configuration file in this format have many
> > quirks. Instead he suggested using XML, JSON or YAML in our example
> > files (so log4j2.xml or log4j2.json or log4j2.yaml) as these are the
> > preferred formats now.
> >
> > David, Viktor and myself have expressed our preference toward adopting
> > YAML for our log4j2 configuration files. I'm wondering if that's
> > enough consensus, or if people want to discuss this further, or even
> > want a vote?
> >
> > Thanks,
> > Mickael
> >
> >
> > On Mon, Nov 18, 2024 at 1:52 PM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> > >
> > > hi, Mickael
> > >
> > > I'm +1 for using the log4j2 format. Otherwise, it's odd for users to
> see
> > > deprecation warnings about log4j.properties when using our example
> files.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > >
> > >
> > > Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午7:33寫道：
> > >
> > > > Hi,
> > > >
> > > > The log4j2 migration PR is pretty much ready to be merged and the
> > > > first deadlines for 4.0.0 is approaching fast.
> > > > I think we should decide which format to use in our example log4j2
> > > > files to avoid having to update the format shortly after 4.0.0.
> > > >
> > > > This point is not really covered by KIP-653: Upgrade log4j to log4j2.
> > > > Do we want another KIP and vote or is a consensus emerging?
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Wed, Nov 6, 2024 at 10:47 AM Mickael Maison <
> > mickael.maison@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I think our example log4j2 file should be as idiomatic as possible.
> > If
> > > > > YAML is now the recommended format, then it makes sense to adopt
> it.
> > I
> > > > > would also do this directly in 4.0.0.
> > > > >
> > > > > Like David, one concern when adding extra dependencies is CVEs.
> YAML
> > > > > is a widely used format and the libraries are actively maintained
> so
> > I
> > > > > think it's acceptable.
> > > > >
> > > > > In my opinion, the configuration format for Kafka is a completely
> > > > > orthogonal issue. If people want to adopt YAML this can be done in
> a
> > > > > separate discussion. I don't see issues with using YAML for
> > > > > configuring log4j2 and properties for Kafka.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Oct 31, 2024 at 5:47 PM Piotr P. Karwasz
> > > > > <pi...@mailing.copernik.eu> wrote:
> > > > > >
> > > > > > Hi Viktor,
> > > > > >
> > > > > > On 31.10.2024 10:19, Viktor Somogyi-Vass wrote:
> > > > > > > I could see a transition for Kafka configs too to YAML. It's
> > widely
> > > > used in
> > > > > > > other projects as well and for Kafka too it would make sense to
> > > > group them
> > > > > > > for instance by log, storage, networking, security etc.. It
> > > > definitely
> > > > > > > has an advantage that we could move to a more well defined
> > structure
> > > > and
> > > > > > > away from these long prefixes which are very cumbersome in some
> > > > places.
> > > > > > > Perhaps 4.0 would have been a good candidate but given that we
> > should
> > > > > > > provide backward compatibility anyway, later is good too.
> > > > > >
> > > > > > To migrate the configuration from Java Properties to YAML and
> > maintain
> > > > > > BC, you can use `jackson-dataformat-properties`. This
> definitively
> > can
> > > > > > be done in a minor release.
> > > > > >
> > > > > > [1]
> > > > > >
> > > >
> >
> https://github.com/FasterXML/jackson-dataformats-text/blob/master/properties/README.md
> > > > > >
> > > > > > > 4. Data bindings and parsers are common sources of CVEs. It
> looks
> > > > like
> > > > > > > Snakeyaml is no exception (
> > > > > > > https://www.cvedetails.com/version-list/0/66013/1/), though it
> > > > doesn't look
> > > > > > > much worse than Jackson. Just to point out, this will add a bit
> > of
> > > > > > > dependency overhead as we keep up with security patches.
> > > > > > >
> > > > > > > It's a good point about CVEs. I haven't seen it on the dev list
> > but
> > > > were
> > > > > > > there any conversations about enabling dependabot version
> > updates?
> > > > With
> > > > > > > automatic dependabot PRs we could get fixes in as soon as it
> > opens
> > > > the PR.
> > > > > >
> > > > > > I don't know how it works in Gradle, but with Maven we don't get
> > > > > > Dependabot PRs for transitive dependencies such as SnakeYAML.
> > Since the
> > > > > > dependency is not present in the `pom.xml`, its version can not
> be
> > > > > > updated. Dependabot might be able to create alerts in the
> > "Security"
> > > > tab
> > > > > > though.
> > > > > >
> > > > > > Since Log4j is a library without an executable distribution, it
> is
> > not
> > > > > > guaranteed that a new version of Log4j will be released each
> time a
> > > > > > vulnerability in SnakeYAML is discovered and that vulnerability
> is
> > > > > > exploitable in Log4j Core. It does not make much sense to make a
> > new
> > > > > > release just to upgrade a number in `pom.xml`.
> > > > > >
> > > > > > Until VEX-es can be created automatically, the only way I can
> > think of
> > > > > > to help the Kafka team with CVEs in transitive dependencies is a
> > manual
> > > > > > process:
> > > > > >
> > > > > > * You mark `jackson-dataformat-yaml` with a comment like "Used by
> > Log4j
> > > > > > Core".
> > > > > >
> > > > > > * If a CVE is reported against that artifact or its dependencies,
> > you
> > > > > > open an issue in Log4j[2], so we can advise on the exploitability
> > of
> > > > the
> > > > > > CVE.
> > > > > >
> > > > > > * With that information you can decide whether to release a new
> > Kafka
> > > > > > version or not.
> > > > > >
> > > > > > Piotr
> > > > > >
> > > > > > [2] https://github.com/apache/logging-log4j2/issues
> > > > > >
> > > > > >
> > > >
> >
>
Re: Configuration format for Log4j Core 2.x
Posted by TengYao Chi <ki...@gmail.com>.
Hi everyone,
From my experience, it has been quite challenging to understand and
transform the .properties file from log4j 1 to log4j 2, as the properties
format is fairly flat and difficult to understand.
In comparison, YAML is more structured, easier to read, and widely used. I
would like to +1 for using YAML properties instead.
If a KIP is required, I am willing to prepare one.
Best,
TengYao
Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午9:35寫道：
> Hi Chia-Ping,
>
> Yes in 4.0.0 we will only provide log4j2 configuration files. We will
> still support log4j.properties files for compatibility but that will
> print a warning when these are used.
>
> However my question is about the format of our log4j2 files. Currently
> in the PR we are using the "properties" format (log4j2.properties).
> But as pointed by Piotr (from the Apache Logging PMC), log4j2 has a
> hierarchical structure which does not translate well into the
> properties format, so configuration file in this format have many
> quirks. Instead he suggested using XML, JSON or YAML in our example
> files (so log4j2.xml or log4j2.json or log4j2.yaml) as these are the
> preferred formats now.
>
> David, Viktor and myself have expressed our preference toward adopting
> YAML for our log4j2 configuration files. I'm wondering if that's
> enough consensus, or if people want to discuss this further, or even
> want a vote?
>
> Thanks,
> Mickael
>
>
> On Mon, Nov 18, 2024 at 1:52 PM Chia-Ping Tsai <ch...@gmail.com> wrote:
> >
> > hi, Mickael
> >
> > I'm +1 for using the log4j2 format. Otherwise, it's odd for users to see
> > deprecation warnings about log4j.properties when using our example files.
> >
> > Best,
> > Chia-Ping
> >
> >
> >
> > Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午7:33寫道：
> >
> > > Hi,
> > >
> > > The log4j2 migration PR is pretty much ready to be merged and the
> > > first deadlines for 4.0.0 is approaching fast.
> > > I think we should decide which format to use in our example log4j2
> > > files to avoid having to update the format shortly after 4.0.0.
> > >
> > > This point is not really covered by KIP-653: Upgrade log4j to log4j2.
> > > Do we want another KIP and vote or is a consensus emerging?
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Wed, Nov 6, 2024 at 10:47 AM Mickael Maison <
> mickael.maison@gmail.com>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > I think our example log4j2 file should be as idiomatic as possible.
> If
> > > > YAML is now the recommended format, then it makes sense to adopt it.
> I
> > > > would also do this directly in 4.0.0.
> > > >
> > > > Like David, one concern when adding extra dependencies is CVEs. YAML
> > > > is a widely used format and the libraries are actively maintained so
> I
> > > > think it's acceptable.
> > > >
> > > > In my opinion, the configuration format for Kafka is a completely
> > > > orthogonal issue. If people want to adopt YAML this can be done in a
> > > > separate discussion. I don't see issues with using YAML for
> > > > configuring log4j2 and properties for Kafka.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Oct 31, 2024 at 5:47 PM Piotr P. Karwasz
> > > > <pi...@mailing.copernik.eu> wrote:
> > > > >
> > > > > Hi Viktor,
> > > > >
> > > > > On 31.10.2024 10:19, Viktor Somogyi-Vass wrote:
> > > > > > I could see a transition for Kafka configs too to YAML. It's
> widely
> > > used in
> > > > > > other projects as well and for Kafka too it would make sense to
> > > group them
> > > > > > for instance by log, storage, networking, security etc.. It
> > > definitely
> > > > > > has an advantage that we could move to a more well defined
> structure
> > > and
> > > > > > away from these long prefixes which are very cumbersome in some
> > > places.
> > > > > > Perhaps 4.0 would have been a good candidate but given that we
> should
> > > > > > provide backward compatibility anyway, later is good too.
> > > > >
> > > > > To migrate the configuration from Java Properties to YAML and
> maintain
> > > > > BC, you can use `jackson-dataformat-properties`. This definitively
> can
> > > > > be done in a minor release.
> > > > >
> > > > > [1]
> > > > >
> > >
> https://github.com/FasterXML/jackson-dataformats-text/blob/master/properties/README.md
> > > > >
> > > > > > 4. Data bindings and parsers are common sources of CVEs. It looks
> > > like
> > > > > > Snakeyaml is no exception (
> > > > > > https://www.cvedetails.com/version-list/0/66013/1/), though it
> > > doesn't look
> > > > > > much worse than Jackson. Just to point out, this will add a bit
> of
> > > > > > dependency overhead as we keep up with security patches.
> > > > > >
> > > > > > It's a good point about CVEs. I haven't seen it on the dev list
> but
> > > were
> > > > > > there any conversations about enabling dependabot version
> updates?
> > > With
> > > > > > automatic dependabot PRs we could get fixes in as soon as it
> opens
> > > the PR.
> > > > >
> > > > > I don't know how it works in Gradle, but with Maven we don't get
> > > > > Dependabot PRs for transitive dependencies such as SnakeYAML.
> Since the
> > > > > dependency is not present in the `pom.xml`, its version can not be
> > > > > updated. Dependabot might be able to create alerts in the
> "Security"
> > > tab
> > > > > though.
> > > > >
> > > > > Since Log4j is a library without an executable distribution, it is
> not
> > > > > guaranteed that a new version of Log4j will be released each time a
> > > > > vulnerability in SnakeYAML is discovered and that vulnerability is
> > > > > exploitable in Log4j Core. It does not make much sense to make a
> new
> > > > > release just to upgrade a number in `pom.xml`.
> > > > >
> > > > > Until VEX-es can be created automatically, the only way I can
> think of
> > > > > to help the Kafka team with CVEs in transitive dependencies is a
> manual
> > > > > process:
> > > > >
> > > > > * You mark `jackson-dataformat-yaml` with a comment like "Used by
> Log4j
> > > > > Core".
> > > > >
> > > > > * If a CVE is reported against that artifact or its dependencies,
> you
> > > > > open an issue in Log4j[2], so we can advise on the exploitability
> of
> > > the
> > > > > CVE.
> > > > >
> > > > > * With that information you can decide whether to release a new
> Kafka
> > > > > version or not.
> > > > >
> > > > > Piotr
> > > > >
> > > > > [2] https://github.com/apache/logging-log4j2/issues
> > > > >
> > > > >
> > >
>
Re: Configuration format for Log4j Core 2.x
Posted by Mickael Maison <mi...@gmail.com>.
Hi Chia-Ping,
Yes in 4.0.0 we will only provide log4j2 configuration files. We will
still support log4j.properties files for compatibility but that will
print a warning when these are used.
However my question is about the format of our log4j2 files. Currently
in the PR we are using the "properties" format (log4j2.properties).
But as pointed by Piotr (from the Apache Logging PMC), log4j2 has a
hierarchical structure which does not translate well into the
properties format, so configuration file in this format have many
quirks. Instead he suggested using XML, JSON or YAML in our example
files (so log4j2.xml or log4j2.json or log4j2.yaml) as these are the
preferred formats now.
David, Viktor and myself have expressed our preference toward adopting
YAML for our log4j2 configuration files. I'm wondering if that's
enough consensus, or if people want to discuss this further, or even
want a vote?
Thanks,
Mickael
On Mon, Nov 18, 2024 at 1:52 PM Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> hi, Mickael
>
> I'm +1 for using the log4j2 format. Otherwise, it's odd for users to see
> deprecation warnings about log4j.properties when using our example files.
>
> Best,
> Chia-Ping
>
>
>
> Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午7:33寫道：
>
> > Hi,
> >
> > The log4j2 migration PR is pretty much ready to be merged and the
> > first deadlines for 4.0.0 is approaching fast.
> > I think we should decide which format to use in our example log4j2
> > files to avoid having to update the format shortly after 4.0.0.
> >
> > This point is not really covered by KIP-653: Upgrade log4j to log4j2.
> > Do we want another KIP and vote or is a consensus emerging?
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Nov 6, 2024 at 10:47 AM Mickael Maison <mi...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > I think our example log4j2 file should be as idiomatic as possible. If
> > > YAML is now the recommended format, then it makes sense to adopt it. I
> > > would also do this directly in 4.0.0.
> > >
> > > Like David, one concern when adding extra dependencies is CVEs. YAML
> > > is a widely used format and the libraries are actively maintained so I
> > > think it's acceptable.
> > >
> > > In my opinion, the configuration format for Kafka is a completely
> > > orthogonal issue. If people want to adopt YAML this can be done in a
> > > separate discussion. I don't see issues with using YAML for
> > > configuring log4j2 and properties for Kafka.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Oct 31, 2024 at 5:47 PM Piotr P. Karwasz
> > > <pi...@mailing.copernik.eu> wrote:
> > > >
> > > > Hi Viktor,
> > > >
> > > > On 31.10.2024 10:19, Viktor Somogyi-Vass wrote:
> > > > > I could see a transition for Kafka configs too to YAML. It's widely
> > used in
> > > > > other projects as well and for Kafka too it would make sense to
> > group them
> > > > > for instance by log, storage, networking, security etc.. It
> > definitely
> > > > > has an advantage that we could move to a more well defined structure
> > and
> > > > > away from these long prefixes which are very cumbersome in some
> > places.
> > > > > Perhaps 4.0 would have been a good candidate but given that we should
> > > > > provide backward compatibility anyway, later is good too.
> > > >
> > > > To migrate the configuration from Java Properties to YAML and maintain
> > > > BC, you can use `jackson-dataformat-properties`. This definitively can
> > > > be done in a minor release.
> > > >
> > > > [1]
> > > >
> > https://github.com/FasterXML/jackson-dataformats-text/blob/master/properties/README.md
> > > >
> > > > > 4. Data bindings and parsers are common sources of CVEs. It looks
> > like
> > > > > Snakeyaml is no exception (
> > > > > https://www.cvedetails.com/version-list/0/66013/1/), though it
> > doesn't look
> > > > > much worse than Jackson. Just to point out, this will add a bit of
> > > > > dependency overhead as we keep up with security patches.
> > > > >
> > > > > It's a good point about CVEs. I haven't seen it on the dev list but
> > were
> > > > > there any conversations about enabling dependabot version updates?
> > With
> > > > > automatic dependabot PRs we could get fixes in as soon as it opens
> > the PR.
> > > >
> > > > I don't know how it works in Gradle, but with Maven we don't get
> > > > Dependabot PRs for transitive dependencies such as SnakeYAML. Since the
> > > > dependency is not present in the `pom.xml`, its version can not be
> > > > updated. Dependabot might be able to create alerts in the "Security"
> > tab
> > > > though.
> > > >
> > > > Since Log4j is a library without an executable distribution, it is not
> > > > guaranteed that a new version of Log4j will be released each time a
> > > > vulnerability in SnakeYAML is discovered and that vulnerability is
> > > > exploitable in Log4j Core. It does not make much sense to make a new
> > > > release just to upgrade a number in `pom.xml`.
> > > >
> > > > Until VEX-es can be created automatically, the only way I can think of
> > > > to help the Kafka team with CVEs in transitive dependencies is a manual
> > > > process:
> > > >
> > > > * You mark `jackson-dataformat-yaml` with a comment like "Used by Log4j
> > > > Core".
> > > >
> > > > * If a CVE is reported against that artifact or its dependencies, you
> > > > open an issue in Log4j[2], so we can advise on the exploitability of
> > the
> > > > CVE.
> > > >
> > > > * With that information you can decide whether to release a new Kafka
> > > > version or not.
> > > >
> > > > Piotr
> > > >
> > > > [2] https://github.com/apache/logging-log4j2/issues
> > > >
> > > >
> >
Re: Configuration format for Log4j Core 2.x
Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi, Mickael
I'm +1 for using the log4j2 format. Otherwise, it's odd for users to see
deprecation warnings about log4j.properties when using our example files.
Best,
Chia-Ping
Mickael Maison <mi...@gmail.com> 於 2024年11月18日 週一 下午7:33寫道：
> Hi,
>
> The log4j2 migration PR is pretty much ready to be merged and the
> first deadlines for 4.0.0 is approaching fast.
> I think we should decide which format to use in our example log4j2
> files to avoid having to update the format shortly after 4.0.0.
>
> This point is not really covered by KIP-653: Upgrade log4j to log4j2.
> Do we want another KIP and vote or is a consensus emerging?
>
> Thanks,
> Mickael
>
> On Wed, Nov 6, 2024 at 10:47 AM Mickael Maison <mi...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I think our example log4j2 file should be as idiomatic as possible. If
> > YAML is now the recommended format, then it makes sense to adopt it. I
> > would also do this directly in 4.0.0.
> >
> > Like David, one concern when adding extra dependencies is CVEs. YAML
> > is a widely used format and the libraries are actively maintained so I
> > think it's acceptable.
> >
> > In my opinion, the configuration format for Kafka is a completely
> > orthogonal issue. If people want to adopt YAML this can be done in a
> > separate discussion. I don't see issues with using YAML for
> > configuring log4j2 and properties for Kafka.
> >
> > Thanks,
> > Mickael
> >
> >
> >
> >
> >
> > On Thu, Oct 31, 2024 at 5:47 PM Piotr P. Karwasz
> > <pi...@mailing.copernik.eu> wrote:
> > >
> > > Hi Viktor,
> > >
> > > On 31.10.2024 10:19, Viktor Somogyi-Vass wrote:
> > > > I could see a transition for Kafka configs too to YAML. It's widely
> used in
> > > > other projects as well and for Kafka too it would make sense to
> group them
> > > > for instance by log, storage, networking, security etc.. It
> definitely
> > > > has an advantage that we could move to a more well defined structure
> and
> > > > away from these long prefixes which are very cumbersome in some
> places.
> > > > Perhaps 4.0 would have been a good candidate but given that we should
> > > > provide backward compatibility anyway, later is good too.
> > >
> > > To migrate the configuration from Java Properties to YAML and maintain
> > > BC, you can use `jackson-dataformat-properties`. This definitively can
> > > be done in a minor release.
> > >
> > > [1]
> > >
> https://github.com/FasterXML/jackson-dataformats-text/blob/master/properties/README.md
> > >
> > > > 4. Data bindings and parsers are common sources of CVEs. It looks
> like
> > > > Snakeyaml is no exception (
> > > > https://www.cvedetails.com/version-list/0/66013/1/), though it
> doesn't look
> > > > much worse than Jackson. Just to point out, this will add a bit of
> > > > dependency overhead as we keep up with security patches.
> > > >
> > > > It's a good point about CVEs. I haven't seen it on the dev list but
> were
> > > > there any conversations about enabling dependabot version updates?
> With
> > > > automatic dependabot PRs we could get fixes in as soon as it opens
> the PR.
> > >
> > > I don't know how it works in Gradle, but with Maven we don't get
> > > Dependabot PRs for transitive dependencies such as SnakeYAML. Since the
> > > dependency is not present in the `pom.xml`, its version can not be
> > > updated. Dependabot might be able to create alerts in the "Security"
> tab
> > > though.
> > >
> > > Since Log4j is a library without an executable distribution, it is not
> > > guaranteed that a new version of Log4j will be released each time a
> > > vulnerability in SnakeYAML is discovered and that vulnerability is
> > > exploitable in Log4j Core. It does not make much sense to make a new
> > > release just to upgrade a number in `pom.xml`.
> > >
> > > Until VEX-es can be created automatically, the only way I can think of
> > > to help the Kafka team with CVEs in transitive dependencies is a manual
> > > process:
> > >
> > > * You mark `jackson-dataformat-yaml` with a comment like "Used by Log4j
> > > Core".
> > >
> > > * If a CVE is reported against that artifact or its dependencies, you
> > > open an issue in Log4j[2], so we can advise on the exploitability of
> the
> > > CVE.
> > >
> > > * With that information you can decide whether to release a new Kafka
> > > version or not.
> > >
> > > Piotr
> > >
> > > [2] https://github.com/apache/logging-log4j2/issues
> > >
> > >
>
Re: Configuration format for Log4j Core 2.x
Posted by Mickael Maison <mi...@gmail.com>.
Hi,
The log4j2 migration PR is pretty much ready to be merged and the
first deadlines for 4.0.0 is approaching fast.
I think we should decide which format to use in our example log4j2
files to avoid having to update the format shortly after 4.0.0.
This point is not really covered by KIP-653: Upgrade log4j to log4j2.
Do we want another KIP and vote or is a consensus emerging?
Thanks,
Mickael
On Wed, Nov 6, 2024 at 10:47 AM Mickael Maison <mi...@gmail.com> wrote:
>
> Hi,
>
> I think our example log4j2 file should be as idiomatic as possible. If
> YAML is now the recommended format, then it makes sense to adopt it. I
> would also do this directly in 4.0.0.
>
> Like David, one concern when adding extra dependencies is CVEs. YAML
> is a widely used format and the libraries are actively maintained so I
> think it's acceptable.
>
> In my opinion, the configuration format for Kafka is a completely
> orthogonal issue. If people want to adopt YAML this can be done in a
> separate discussion. I don't see issues with using YAML for
> configuring log4j2 and properties for Kafka.
>
> Thanks,
> Mickael
>
>
>
>
>
> On Thu, Oct 31, 2024 at 5:47 PM Piotr P. Karwasz
> <pi...@mailing.copernik.eu> wrote:
> >
> > Hi Viktor,
> >
> > On 31.10.2024 10:19, Viktor Somogyi-Vass wrote:
> > > I could see a transition for Kafka configs too to YAML. It's widely used in
> > > other projects as well and for Kafka too it would make sense to group them
> > > for instance by log, storage, networking, security etc.. It definitely
> > > has an advantage that we could move to a more well defined structure and
> > > away from these long prefixes which are very cumbersome in some places.
> > > Perhaps 4.0 would have been a good candidate but given that we should
> > > provide backward compatibility anyway, later is good too.
> >
> > To migrate the configuration from Java Properties to YAML and maintain
> > BC, you can use `jackson-dataformat-properties`. This definitively can
> > be done in a minor release.
> >
> > [1]
> > https://github.com/FasterXML/jackson-dataformats-text/blob/master/properties/README.md
> >
> > > 4. Data bindings and parsers are common sources of CVEs. It looks like
> > > Snakeyaml is no exception (
> > > https://www.cvedetails.com/version-list/0/66013/1/), though it doesn't look
> > > much worse than Jackson. Just to point out, this will add a bit of
> > > dependency overhead as we keep up with security patches.
> > >
> > > It's a good point about CVEs. I haven't seen it on the dev list but were
> > > there any conversations about enabling dependabot version updates? With
> > > automatic dependabot PRs we could get fixes in as soon as it opens the PR.
> >
> > I don't know how it works in Gradle, but with Maven we don't get
> > Dependabot PRs for transitive dependencies such as SnakeYAML. Since the
> > dependency is not present in the `pom.xml`, its version can not be
> > updated. Dependabot might be able to create alerts in the "Security" tab
> > though.
> >
> > Since Log4j is a library without an executable distribution, it is not
> > guaranteed that a new version of Log4j will be released each time a
> > vulnerability in SnakeYAML is discovered and that vulnerability is
> > exploitable in Log4j Core. It does not make much sense to make a new
> > release just to upgrade a number in `pom.xml`.
> >
> > Until VEX-es can be created automatically, the only way I can think of
> > to help the Kafka team with CVEs in transitive dependencies is a manual
> > process:
> >
> > * You mark `jackson-dataformat-yaml` with a comment like "Used by Log4j
> > Core".
> >
> > * If a CVE is reported against that artifact or its dependencies, you
> > open an issue in Log4j[2], so we can advise on the exploitability of the
> > CVE.
> >
> > * With that information you can decide whether to release a new Kafka
> > version or not.
> >
> > Piotr
> >
> > [2] https://github.com/apache/logging-log4j2/issues
> >
> >