You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by David Kim <da...@braintreepayments.com> on 2016/06/01 16:42:43 UTC
Flink Kafka Unused Properties Log Warnings
Hello!
Using Flink 1.0.3.
This is cosmetic but will help clean up logging I think.
The Apache *KafkaConsumer* logs a warning [1] for any unused properties.
This is great in case the developer has a typo or should clean up any
unused keys.
Flink's Kafka consumer and producer have some custom properties and it
passes the entire properties map into the Apache Kafka consumer and
producer. Custom properties such as "flink.poll-timeout" make it through
and it triggers a warning.
For the sake of cleaning up logging and removing false positives, I think
it might make sense for the FlinkKafkaConsumer09 [2] and
FlinkKafkaProducerBase [3] to filter out custom flink properties.
Let me know if this sounds reasonable!
Thanks!
David
[1]
https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L639
[2]
https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java#L186
[3]
https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L153
Re: Flink Kafka Unused Properties Log Warnings
Posted by David Kim <da...@braintreepayments.com>.
Awesome, thanks Robert!
On Wed, Jun 1, 2016 at 1:12 PM Robert Metzger <rm...@apache.org> wrote:
> Hi Kim,
>
> I agree with you. We should definitively filter out custom Flink
> properties. I've filed a JIRA for this:
> https://issues.apache.org/jira/browse/FLINK-4004
> Thanks for the report!
>
> Regards,
> Robert
>
>
> On Wed, Jun 1, 2016 at 6:42 PM, David Kim <david.kim@braintreepayments.com
> > wrote:
>
>> Hello!
>>
>> Using Flink 1.0.3.
>>
>> This is cosmetic but will help clean up logging I think.
>>
>> The Apache *KafkaConsumer* logs a warning [1] for any unused properties.
>> This is great in case the developer has a typo or should clean up any
>> unused keys.
>>
>> Flink's Kafka consumer and producer have some custom properties and it
>> passes the entire properties map into the Apache Kafka consumer and
>> producer. Custom properties such as "flink.poll-timeout" make it through
>> and it triggers a warning.
>>
>> For the sake of cleaning up logging and removing false positives, I think
>> it might make sense for the FlinkKafkaConsumer09 [2] and
>> FlinkKafkaProducerBase [3] to filter out custom flink properties.
>>
>> Let me know if this sounds reasonable!
>>
>> Thanks!
>> David
>>
>> [1]
>> https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L639
>> [2]
>> https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java#L186
>> [3]
>> https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L153
>>
>>
>>
>
Re: Flink Kafka Unused Properties Log Warnings
Posted by Robert Metzger <rm...@apache.org>.
Hi Kim,
I agree with you. We should definitively filter out custom Flink
properties. I've filed a JIRA for this:
https://issues.apache.org/jira/browse/FLINK-4004
Thanks for the report!
Regards,
Robert
On Wed, Jun 1, 2016 at 6:42 PM, David Kim <da...@braintreepayments.com>
wrote:
> Hello!
>
> Using Flink 1.0.3.
>
> This is cosmetic but will help clean up logging I think.
>
> The Apache *KafkaConsumer* logs a warning [1] for any unused properties.
> This is great in case the developer has a typo or should clean up any
> unused keys.
>
> Flink's Kafka consumer and producer have some custom properties and it
> passes the entire properties map into the Apache Kafka consumer and
> producer. Custom properties such as "flink.poll-timeout" make it through
> and it triggers a warning.
>
> For the sake of cleaning up logging and removing false positives, I think
> it might make sense for the FlinkKafkaConsumer09 [2] and
> FlinkKafkaProducerBase [3] to filter out custom flink properties.
>
> Let me know if this sounds reasonable!
>
> Thanks!
> David
>
> [1]
> https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L639
> [2]
> https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.java#L186
> [3]
> https://github.com/apache/flink/blob/release-1.0.3-rc3/flink-streaming-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducerBase.java#L153
>
>
>