You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Valeria Vasylieva (JIRA)" <ji...@apache.org> on 2019/03/19 11:31:00 UTC
[jira] [Comment Edited] (KAFKA-6675) Connect workers should log
plugin path and available plugins more clearly
[ https://issues.apache.org/jira/browse/KAFKA-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795993#comment-16795993 ]
Valeria Vasylieva edited comment on KAFKA-6675 at 3/19/19 11:30 AM:
--------------------------------------------------------------------
[~rhauch] I have investigated the actual log:
if we have such configuration:
{code:java}
plugin.path=/etc/kafka/kafka_2.11-2.1.0/connect-plugins, /etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium
{code}
then, we have such output for plugin.path plugins:
{code:java}
[2019-03-19 13:32:47,277] INFO Registered loader: PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-cassandra-1.0.2/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,278] INFO Added plugin 'io.confluent.connect.cassandra.CassandraSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,279] INFO Loading plugin from: /etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:47,694] INFO Registered loader: PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,694] INFO Added plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,695] INFO Loading plugin from: /etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:48,083] INFO Registered loader: PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:48,090] INFO Added plugin 'io.debezium.connector.mysql.MySqlConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,128] INFO Registered loader: sun.misc.Launcher$AppClassLoader@764c12b6 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:50,131] INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,161] INFO Added aliases 'CassandraSinkConnector' and 'CassandraSink' to plugin 'io.confluent.connect.cassandra.CassandraSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,161] INFO Added aliases 'MySqlConnector' and 'MySql' to plugin 'io.debezium.connector.mysql.MySqlConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390){code}
and such output for classpath plugins:
{code:java}
[2019-03-19 13:32:50,131] INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,132] INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,163] INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,164] INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
{code}
What is the goal of this task, to make the output more clear and pretty? Almost all the information (including path), that you stated is already available in the log.
was (Author: nimfadora):
[~rhauch] I have investigated the actual log:
if we have such configuration:
{code:java}
plugin.path=/etc/kafka/kafka_2.11-2.1.0/connect-plugins, /etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium
{code}
then, we have such output for plugin.path plugins:
{code:java}
[2019-03-19 13:32:47,277] INFO Registered loader: PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-cassandra-1.0.2/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,278] INFO Added plugin 'io.confluent.connect.cassandra.CassandraSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,279] INFO Loading plugin from: /etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:47,694] INFO Registered loader: PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,694] INFO Added plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,695] INFO Loading plugin from: /etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:48,083] INFO Registered loader: PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:48,090] INFO Added plugin 'io.debezium.connector.mysql.MySqlConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,128] INFO Registered loader: sun.misc.Launcher$AppClassLoader@764c12b6 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:50,131] INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,161] INFO Added aliases 'CassandraSinkConnector' and 'CassandraSink' to plugin 'io.confluent.connect.cassandra.CassandraSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,161] INFO Added aliases 'MySqlConnector' and 'MySql' to plugin 'io.debezium.connector.mysql.MySqlConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390){code}
and such output for classpath plugins:
{code:java}
[2019-03-19 13:32:50,131] INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,132] INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,163] INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,164] INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
{code}
What is the goal of this task, to make the output more clear and pretty? Almost all the information (including path), that you stated is already available in the log.
> Connect workers should log plugin path and available plugins more clearly
> -------------------------------------------------------------------------
>
> Key: KAFKA-6675
> URL: https://issues.apache.org/jira/browse/KAFKA-6675
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 0.11.0.1
> Reporter: Randall Hauch
> Assignee: Valeria Vasylieva
> Priority: Minor
>
> Users struggle with setting the plugin path and properly installing plugins. If users get any of this wrong, they get strange errors only after they run the worker and attempt to deploy connectors or use transformations.
> The Connect worker should more obviously output the plugin path directories and the available plugins. For example, if the {{plugin.path}} were:
> {code}
> plugin.path=/usr/local/share/java,/usr/local/plugins
> {code}
> then the worker might output something like the following information in the log:
> {noformat}
> Looking for plugins on classpath and inside plugin.path directories:
> /usr/local/share/java
> /usr/local/plugins
>
> Source Connector(s):
> FileStreamSource (org.apache.kafka.connect.file.FileStreamSourceConnector) @ classpath
> FileStreamSink (org.apache.kafka.connect.file.FileStreamSinkConnector) @ classpath
> JdbcSource (io.confluent.connect.jdbc.JdbcSourceConnector) @ /usr/local/share/java/kafka-connect-jdbc
> MySql (io.debezium.connector.mysql.MySqlConnector) @ /usr/local/plugins/debezium-connector-mysql
> Converter(s):
> JsonConverter (org.apache.kafka.connect.json.JsonConverter) @ classpath
> ByteArrayConverter (org.apache.kafka.connect.converters.ByteArrayConverter) @ classpath
> SimpleHeaderConverter (org.apache.kafka.connect.converters.SimpleHeaderConverter) @ classpath
> AvroConverter (io.confluent.connect.avro.AvroConverter) @ /usr/local/share/java/kafka-serde-tools
> Transformation(s):
> InsertField (org.apache.kafka.connect.transforms.InsertField) @ classpath
> ReplaceField (org.apache.kafka.connect.transforms.ReplaceField) @ classpath
> MaskField (org.apache.kafka.connect.transforms.MaskField) @ classpath
> ValueToKey (org.apache.kafka.connect.transforms.ValueToKey) @ classpath
> HoistField (org.apache.kafka.connect.transforms.HoistField) @ classpath
> ExtractField (org.apache.kafka.connect.transforms.ExtractField) @ classpath
> SetSchemaMetadata (org.apache.kafka.connect.transforms.SetSchemaMetadata) @ classpath
> RegexRouter (org.apache.kafka.connect.transforms.RegexRouter) @ classpath
> TimestampRouter (org.apache.kafka.connect.transforms.TimestampRouter) @ classpath
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)