How can I specify a Java class in the kafka bolt Producer Config?

["Aaron.Dossett" <Aa...@target.com>]

Kafka 0.9 requires that the "key.serializer" and "value.serializer" items in ProducerConfig be java classes, not a string containing the name of a java class.
See https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L280-L287

However, if try to do that like this:

Properties props = new Properties();props.put("bootstrap.servers", topoProperties.getProperty("bootstrap.servers"));
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class);

conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props);
Storm fails to start with this error:

java.lang.IllegalArgumentException: Topology conf is not json-serializable
See: https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/StormSubmitter.java#L192-L194

Seems like kafka-bolt's prepare method will have to transform a string into a Java class. Before I go down that path, I was wondering if anyone else had run into this problem and if there's a workaround?

Thanks! -Aaron

Re: How can I specify a Java class in the kafka bolt Producer Config?

["Aaron.Dossett" <Aa...@target.com>]

I should have done a bit more research... this is a problem in Storm 0.10, but it looks like the upcoming Storm 1.0 release will handle this situation by allowing properties to be passed directly to the Kafka Bolt instead of indirectly through the Topology configuration.

If there's going to be a Storm 0.10.1 release, this could be a helpful bug fix.

From: Aaron Dossett <Aa...@target.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Friday, March 11, 2016 at 1:19 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: How can I specify a Java class in the kafka bolt Producer Config?

Kafka 0.9 requires that the "key.serializer" and "value.serializer" items in ProducerConfig be java classes, not a string containing the name of a java class.
See https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L280-L287

However, if try to do that like this:

Properties props = new Properties();props.put("bootstrap.servers", topoProperties.getProperty("bootstrap.servers"));
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class);

conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props);
Storm fails to start with this error:

java.lang.IllegalArgumentException: Topology conf is not json-serializable
See: https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/StormSubmitter.java#L192-L194

Seems like kafka-bolt's prepare method will have to transform a string into a Java class. Before I go down that path, I was wondering if anyone else had run into this problem and if there's a workaround?

Thanks! -Aaron