You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jaikiran Pai <ja...@gmail.com> on 2016/08/16 08:58:22 UTC
Do not log value of configs that Kafka doesn't recognize
We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
and producers. In one of our consumers, our consumer config had a SSL
specific property which ended up being used against a non-SSL Kafka
broker port. As a result, the logs ended up seeing messages like:
17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration
*ssl.truststore.password = foobar* was supplied but isn't a known config.
The log message is fine and makes sense, but can Kafka please not log
the values of the properties and instead just include the config name
which it considers as unknown? That way it won't ended up logging these
potentially sensitive values. I understand that only those with access
to these log files can end up seeing these values but even then some of
our internal processes forbid logging such sensitive information to the
logs. This log message will still end up being useful if only the config
name is logged without the value.
Can I add this as a JIRA and provide a patch?
-Jaikiran
Re: Do not log value of configs that Kafka doesn't recognize
Posted by Jaikiran Pai <ja...@gmail.com>.
Created https://issues.apache.org/jira/browse/KAFKA-4056.
-Jaikiran
On Wednesday 17 August 2016 06:28 PM, Ismael Juma wrote:
> Yes, please file a JIRA.
>
> Thanks,
> Ismael
>
> On Wed, Aug 17, 2016 at 1:46 PM, Jaikiran Pai <ja...@gmail.com>
> wrote:
>
>> Thanks for the inputs.
>>
>> I think it's fine if Kafka selectively logs values for certain configs and
>> masks others (the passwords). So from what I understand, this looks more
>> like a bug where it unintentionally ended up logging these values. Do you
>> want me to file a JIRA for this?
>>
>> -Jaikiran
>>
>>
>>
>> On Wednesday 17 August 2016 06:09 PM, Ismael Juma wrote:
>>
>>> Note that we already mask password related config values[1]. The issue is
>>> that it only happens after parsing, and the output of `logUnused` is based
>>> on the `originals` map, which has the values before parsing:
>>>
>>> public void logUnused() {
>>> for (String key : unused())
>>> log.warn("The configuration {} = {} was supplied but isn't a
>>> known config.", key, this.originals.get(key));
>>> }
>>>
>>> Ismael
>>>
>>> [1]
>>> https://github.com/apache/kafka/commit/ab5ac264a71d7f895b21b
>>> 4acfd93d9581dabd7c1
>>>
>>> On Wed, Aug 17, 2016 at 12:55 PM, Manikumar Reddy <
>>> manikumar.reddy@gmail.com
>>>
>>>> wrote:
>>>> During server/client startup, we are logging all the supplied configs.
>>>> May
>>>> be we can just mask
>>>> the password related config values for both valid/invalid configs.
>>>>
>>>> On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai <ja...@gmail.com>
>>>> wrote:
>>>>
>>>> Any opinion about this proposed change?
>>>>> -Jaikiran
>>>>>
>>>>> On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
>>>>>
>>>>> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
>>>>>> and producers. In one of our consumers, our consumer config had a SSL
>>>>>> specific property which ended up being used against a non-SSL Kafka
>>>>>>
>>>>> broker
>>>>> port. As a result, the logs ended up seeing messages like:
>>>>>> 17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration
>>>>>> *ssl.truststore.password = foobar* was supplied but isn't a known
>>>>>>
>>>>> config.
>>>>> The log message is fine and makes sense, but can Kafka please not log
>>>>> the
>>>>> values of the properties and instead just include the config name which
>>>>> it
>>>>> considers as unknown? That way it won't ended up logging these
>>>>> potentially
>>>>> sensitive values. I understand that only those with access to these log
>>>>>> files can end up seeing these values but even then some of our internal
>>>>>> processes forbid logging such sensitive information to the logs. This
>>>>>>
>>>>> log
>>>>> message will still end up being useful if only the config name is logged
>>>>>> without the value.
>>>>>>
>>>>>> Can I add this as a JIRA and provide a patch?
>>>>>>
>>>>>> -Jaikiran
>>>>>>
>>>>>>
Re: Do not log value of configs that Kafka doesn't recognize
Posted by Ismael Juma <is...@juma.me.uk>.
Yes, please file a JIRA.
Thanks,
Ismael
On Wed, Aug 17, 2016 at 1:46 PM, Jaikiran Pai <ja...@gmail.com>
wrote:
> Thanks for the inputs.
>
> I think it's fine if Kafka selectively logs values for certain configs and
> masks others (the passwords). So from what I understand, this looks more
> like a bug where it unintentionally ended up logging these values. Do you
> want me to file a JIRA for this?
>
> -Jaikiran
>
>
>
> On Wednesday 17 August 2016 06:09 PM, Ismael Juma wrote:
>
>> Note that we already mask password related config values[1]. The issue is
>> that it only happens after parsing, and the output of `logUnused` is based
>> on the `originals` map, which has the values before parsing:
>>
>> public void logUnused() {
>> for (String key : unused())
>> log.warn("The configuration {} = {} was supplied but isn't a
>> known config.", key, this.originals.get(key));
>> }
>>
>> Ismael
>>
>> [1]
>> https://github.com/apache/kafka/commit/ab5ac264a71d7f895b21b
>> 4acfd93d9581dabd7c1
>>
>> On Wed, Aug 17, 2016 at 12:55 PM, Manikumar Reddy <
>> manikumar.reddy@gmail.com
>>
>>> wrote:
>>> During server/client startup, we are logging all the supplied configs.
>>> May
>>> be we can just mask
>>> the password related config values for both valid/invalid configs.
>>>
>>> On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai <ja...@gmail.com>
>>> wrote:
>>>
>>> Any opinion about this proposed change?
>>>>
>>>> -Jaikiran
>>>>
>>>> On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
>>>>
>>>> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
>>>>> and producers. In one of our consumers, our consumer config had a SSL
>>>>> specific property which ended up being used against a non-SSL Kafka
>>>>>
>>>> broker
>>>
>>>> port. As a result, the logs ended up seeing messages like:
>>>>>
>>>>> 17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration
>>>>> *ssl.truststore.password = foobar* was supplied but isn't a known
>>>>>
>>>> config.
>>>
>>>> The log message is fine and makes sense, but can Kafka please not log
>>>>>
>>>> the
>>>
>>>> values of the properties and instead just include the config name which
>>>>>
>>>> it
>>>
>>>> considers as unknown? That way it won't ended up logging these
>>>>>
>>>> potentially
>>>
>>>> sensitive values. I understand that only those with access to these log
>>>>> files can end up seeing these values but even then some of our internal
>>>>> processes forbid logging such sensitive information to the logs. This
>>>>>
>>>> log
>>>
>>>> message will still end up being useful if only the config name is logged
>>>>> without the value.
>>>>>
>>>>> Can I add this as a JIRA and provide a patch?
>>>>>
>>>>> -Jaikiran
>>>>>
>>>>>
>>>>
>
Re: Do not log value of configs that Kafka doesn't recognize
Posted by Jaikiran Pai <ja...@gmail.com>.
Thanks for the inputs.
I think it's fine if Kafka selectively logs values for certain configs
and masks others (the passwords). So from what I understand, this looks
more like a bug where it unintentionally ended up logging these values.
Do you want me to file a JIRA for this?
-Jaikiran
On Wednesday 17 August 2016 06:09 PM, Ismael Juma wrote:
> Note that we already mask password related config values[1]. The issue is
> that it only happens after parsing, and the output of `logUnused` is based
> on the `originals` map, which has the values before parsing:
>
> public void logUnused() {
> for (String key : unused())
> log.warn("The configuration {} = {} was supplied but isn't a
> known config.", key, this.originals.get(key));
> }
>
> Ismael
>
> [1]
> https://github.com/apache/kafka/commit/ab5ac264a71d7f895b21b4acfd93d9581dabd7c1
>
> On Wed, Aug 17, 2016 at 12:55 PM, Manikumar Reddy <manikumar.reddy@gmail.com
>> wrote:
>> During server/client startup, we are logging all the supplied configs. May
>> be we can just mask
>> the password related config values for both valid/invalid configs.
>>
>> On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai <ja...@gmail.com>
>> wrote:
>>
>>> Any opinion about this proposed change?
>>>
>>> -Jaikiran
>>>
>>> On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
>>>
>>>> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
>>>> and producers. In one of our consumers, our consumer config had a SSL
>>>> specific property which ended up being used against a non-SSL Kafka
>> broker
>>>> port. As a result, the logs ended up seeing messages like:
>>>>
>>>> 17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration
>>>> *ssl.truststore.password = foobar* was supplied but isn't a known
>> config.
>>>> The log message is fine and makes sense, but can Kafka please not log
>> the
>>>> values of the properties and instead just include the config name which
>> it
>>>> considers as unknown? That way it won't ended up logging these
>> potentially
>>>> sensitive values. I understand that only those with access to these log
>>>> files can end up seeing these values but even then some of our internal
>>>> processes forbid logging such sensitive information to the logs. This
>> log
>>>> message will still end up being useful if only the config name is logged
>>>> without the value.
>>>>
>>>> Can I add this as a JIRA and provide a patch?
>>>>
>>>> -Jaikiran
>>>>
>>>
Re: Do not log value of configs that Kafka doesn't recognize
Posted by Ismael Juma <is...@juma.me.uk>.
Note that we already mask password related config values[1]. The issue is
that it only happens after parsing, and the output of `logUnused` is based
on the `originals` map, which has the values before parsing:
public void logUnused() {
for (String key : unused())
log.warn("The configuration {} = {} was supplied but isn't a
known config.", key, this.originals.get(key));
}
Ismael
[1]
https://github.com/apache/kafka/commit/ab5ac264a71d7f895b21b4acfd93d9581dabd7c1
On Wed, Aug 17, 2016 at 12:55 PM, Manikumar Reddy <manikumar.reddy@gmail.com
> wrote:
> During server/client startup, we are logging all the supplied configs. May
> be we can just mask
> the password related config values for both valid/invalid configs.
>
> On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai <ja...@gmail.com>
> wrote:
>
> > Any opinion about this proposed change?
> >
> > -Jaikiran
> >
> > On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
> >
> >> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
> >> and producers. In one of our consumers, our consumer config had a SSL
> >> specific property which ended up being used against a non-SSL Kafka
> broker
> >> port. As a result, the logs ended up seeing messages like:
> >>
> >> 17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration
> >> *ssl.truststore.password = foobar* was supplied but isn't a known
> config.
> >>
> >> The log message is fine and makes sense, but can Kafka please not log
> the
> >> values of the properties and instead just include the config name which
> it
> >> considers as unknown? That way it won't ended up logging these
> potentially
> >> sensitive values. I understand that only those with access to these log
> >> files can end up seeing these values but even then some of our internal
> >> processes forbid logging such sensitive information to the logs. This
> log
> >> message will still end up being useful if only the config name is logged
> >> without the value.
> >>
> >> Can I add this as a JIRA and provide a patch?
> >>
> >> -Jaikiran
> >>
> >
> >
>
Re: Do not log value of configs that Kafka doesn't recognize
Posted by Manikumar Reddy <ma...@gmail.com>.
During server/client startup, we are logging all the supplied configs. May
be we can just mask
the password related config values for both valid/invalid configs.
On Wed, Aug 17, 2016 at 5:14 PM, Jaikiran Pai <ja...@gmail.com>
wrote:
> Any opinion about this proposed change?
>
> -Jaikiran
>
> On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
>
>> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
>> and producers. In one of our consumers, our consumer config had a SSL
>> specific property which ended up being used against a non-SSL Kafka broker
>> port. As a result, the logs ended up seeing messages like:
>>
>> 17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration
>> *ssl.truststore.password = foobar* was supplied but isn't a known config.
>>
>> The log message is fine and makes sense, but can Kafka please not log the
>> values of the properties and instead just include the config name which it
>> considers as unknown? That way it won't ended up logging these potentially
>> sensitive values. I understand that only those with access to these log
>> files can end up seeing these values but even then some of our internal
>> processes forbid logging such sensitive information to the logs. This log
>> message will still end up being useful if only the config name is logged
>> without the value.
>>
>> Can I add this as a JIRA and provide a patch?
>>
>> -Jaikiran
>>
>
>
Re: Do not log value of configs that Kafka doesn't recognize
Posted by Jaikiran Pai <ja...@gmail.com>.
Any opinion about this proposed change?
-Jaikiran
On Tuesday 16 August 2016 02:28 PM, Jaikiran Pai wrote:
> We are using 0.9.0.1 of Kafka (Java) libraries for our Kafka consumers
> and producers. In one of our consumers, our consumer config had a SSL
> specific property which ended up being used against a non-SSL Kafka
> broker port. As a result, the logs ended up seeing messages like:
>
> 17:53:33,722 WARN [o.a.k.c.c.ConsumerConfig] - The configuration
> *ssl.truststore.password = foobar* was supplied but isn't a known config.
>
> The log message is fine and makes sense, but can Kafka please not log
> the values of the properties and instead just include the config name
> which it considers as unknown? That way it won't ended up logging
> these potentially sensitive values. I understand that only those with
> access to these log files can end up seeing these values but even then
> some of our internal processes forbid logging such sensitive
> information to the logs. This log message will still end up being
> useful if only the config name is logged without the value.
>
> Can I add this as a JIRA and provide a patch?
>
> -Jaikiran