You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Mr Kafka (JIRA)" <ji...@apache.org> on 2018/10/22 22:34:00 UTC
[jira] [Comment Edited] (KAFKA-7510) KStreams RecordCollectorImpl
leaks data to logs on error
[ https://issues.apache.org/jira/browse/KAFKA-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659769#comment-16659769 ]
Mr Kafka edited comment on KAFKA-7510 at 10/22/18 10:33 PM:
------------------------------------------------------------
[~mjsax]
{quote}Why does it not contradict KAFKA-6538? It is about adding key/value in human readable form to exception messages that end up in the logs. While this ticket is about removing key/value data from the logs. What do you mean by "it only affects it's implementation"?
{quote}
You can still add key/vaue/headers or as much useful data as you want to the log messages, those log messages will be under TRACE level and a user has to explicitly turn TRACE level on to see the extra information.
Happy to create a PR to move out key/value logging from RecordCollectorImpl to trace as it directly blocks me and I'm likely to do this in a private fork regardless until it is in an official release. Not so interested on creating a KIP and long discussions on moving sensitive log data to a different log level which is as simple as adding a *log.trace(...)* below the *log.error(...)* statements.
was (Author: mrkafka):
[~mjsax]
{quote}
Why does it not contradict KAFKA-6538? It is about adding key/value in human readable form to exception messages that end up in the logs. While this ticket is about removing key/value data from the logs. What do you mean by "it only affects it's implementation"?
{quote}
You can still add key/vaue/headers or as much useful data as you want to the log messages, those log messages will be under TRACE level and a user has to explicitly turn TRACE level on to see the extra information.
Happy to create a PR to move out key/value logging from RecordCollectorImpl to trace as it directly blocks me and I'm likely to do this in a private fork regardless until it is done. Not so interested on creating a KIP and long discussions on moving sensitive log data to a different log level which is as simple as adding a *log.trace(...)* below the *log.error(...)* statements.
> KStreams RecordCollectorImpl leaks data to logs on error
> --------------------------------------------------------
>
> Key: KAFKA-7510
> URL: https://issues.apache.org/jira/browse/KAFKA-7510
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Mr Kafka
> Priority: Major
> Labels: user-experience
>
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl leaks data on error as it dumps the *value* / message payload to the logs.
> This is problematic as it may contain personally identifiable information (pii) or other secret information to plain text log files which can then be propagated to other log systems i.e Splunk.
> I suggest the *key*, and *value* fields be moved to debug level as it is useful for some people while error level contains the *errorMessage, timestamp, topic* and *stackTrace*.
> {code:java}
> private <K, V> void recordSendError(
> final K key,
> final V value,
> final Long timestamp,
> final String topic,
> final Exception exception
> ) {
> String errorLogMessage = LOG_MESSAGE;
> String errorMessage = EXCEPTION_MESSAGE;
> if (exception instanceof RetriableException) {
> errorLogMessage += PARAMETER_HINT;
> errorMessage += PARAMETER_HINT;
> }
> log.error(errorLogMessage, key, value, timestamp, topic, exception.toString());
> sendException = new StreamsException(
> String.format(
> errorMessage,
> logPrefix,
> "an error caught",
> key,
> value,
> timestamp,
> topic,
> exception.toString()
> ),
> exception);
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)