You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Garry Turkington <g....@improvedigital.com> on 2013/11/19 01:53:44 UTC
Problem testing creation of ConsumerConfig object (re SAMZA-77)
Hi,
I thought SAMZA-77 looked like fun so made the changes required to add the stringified UUID to the client and group IDs. All nice and easy. Then I tried adding some test cases and think I'm managing to make this more difficult than it needs to be. :)
In my test case I have the following, which I believe is the right chain of calls to create a KafkaConfig object (I created a simple config file in the resources directory):
val factory = new PropertiesConfigFactory()
val config = factory.getConfig(URI.create("file://%s/src/test/resources/test.properties<file:///\\%25s\src\test\resources\test.properties>" format new File
(".").getCanonicalPath))
val kafkaConfig = new KafkaConfig(config)
All well and good but then I try and call the appropriate method on the KafkaConfig object (with some uncertainty of what the argument needs map to):
val config1 = kafkaConfig.getKafkaSystemConsumerConfig("appname")
This causes an exception to be thrown, the highlights being:
java.lang.IllegalArgumentException: requirement failed: Missing required property 'zookeeper.connect'
at scala.Predef$.require(Predef.scala:145)
at kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:175)
at kafka.utils.ZKConfig.<init>(ZkUtils.scala:775)
at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:73)
at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:77)
at org.apache.samza.config.KafkaConfig.getKafkaSystemConsumerConfig(KafkaConfig.scala:91)
This suggests that to create a ConsumerConfig requires the whole ZK/Kafka stack to be instantiated - is this the case and if so pointers to the minimum config set required to get it working? I tried to piece something together from the other test cases but couldn't find the right magic.
Or is there an easier way to do all this? :)
Thanks
Garry
RE: Problem testing creation of ConsumerConfig object (re SAMZA-77)
Posted by Garry Turkington <g....@improvedigital.com>.
Hi Chris,
Thanks for the pointers, very useful. I got my tests to pass but seem to have broken the TopicMetadataCache in the process; I assume somewhere internally it's using the client id (not group id) as part of an equality test. But I'll dig into it tomorrow.
Thanks,
Garry
-----Original Message-----
From: Chris Riccomini [mailto:criccomini@linkedin.com]
Sent: 19 November 2013 01:07
To: dev@samza.incubator.apache.org
Subject: Re: Problem testing creation of ConsumerConfig object (re SAMZA-77)
Hey Gary,
Wow, great! :)
Kafka, does indeed have some requirements on what config parameters it needs, and it does verify these at instantiation time. The parameters you'll need to set are:
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactor
y
systems.kafka.consumer.zookeeper.connect=localhost:2181/
systems.kafka.producer.metadata.broker.list=localhost:9092
Note that, in this case, the system is called "kafka". In your example, you'd use the "systems.appname.*" prefix (or change your test to call
getKafkaSystemConsumerConfig("kafka") ).
Kafka does come with a way to do this in their TestUtils class:
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/u
tils/TestUtils.scala?source=c
The method you're probably looking for is:
createConsumerProperties
And:
getProducerConfig
What I'd recommend is just creating a java.util.Properties instance, and adding all appropriate configs to it (either with putAll, or put). Then I'd use the MapConfig class to convert the Properties to a Config object:
val config = new MapConfig(myProps);
This is a bit easier than going the factory route that you're on right now, and it can all be done inside your unit test, instead of having extra properties files.
Cheers,
Chris
On 11/18/13 4:53 PM, "Garry Turkington" <g....@improvedigital.com>
wrote:
>Hi,
>
>I thought SAMZA-77 looked like fun so made the changes required to add
>the stringified UUID to the client and group IDs. All nice and easy.
>Then I tried adding some test cases and think I'm managing to make this
>more difficult than it needs to be. :)
>
>In my test case I have the following, which I believe is the right
>chain of calls to create a KafkaConfig object (I created a simple
>config file in the resources directory):
>
> val factory = new PropertiesConfigFactory()
> val config =
>factory.getConfig(URI.create("file://%s/src/test/resources/test.propert
>ies <file:///\\%25s\src\test\resources\test.properties>" format new
>File
>(".").getCanonicalPath))
>val kafkaConfig = new KafkaConfig(config)
>
>All well and good but then I try and call the appropriate method on the
>KafkaConfig object (with some uncertainty of what the argument needs
>map
>to):
>
>val config1 = kafkaConfig.getKafkaSystemConsumerConfig("appname")
>
>This causes an exception to be thrown, the highlights being:
>
>java.lang.IllegalArgumentException: requirement failed: Missing
>required property 'zookeeper.connect'
> at scala.Predef$.require(Predef.scala:145)
> at
>kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:175)
> at kafka.utils.ZKConfig.<init>(ZkUtils.scala:775)
> at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:73)
> at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:77)
> at
>org.apache.samza.config.KafkaConfig.getKafkaSystemConsumerConfig(KafkaC
>onf
>ig.scala:91)
>
>This suggests that to create a ConsumerConfig requires the whole
>ZK/Kafka stack to be instantiated - is this the case and if so pointers
>to the minimum config set required to get it working? I tried to piece
>something together from the other test cases but couldn't find the
>right magic.
>
>Or is there an easier way to do all this? :)
>
>Thanks
>Garry
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2013.0.3426 / Virus Database: 3629/6841 - Release Date: 11/16/13
Re: Problem testing creation of ConsumerConfig object (re SAMZA-77)
Posted by Chris Riccomini <cr...@linkedin.com>.
Hey Gary,
Wow, great! :)
Kafka, does indeed have some requirements on what config parameters it
needs, and it does verify these at instantiation time. The parameters
you'll need to set are:
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactor
y
systems.kafka.consumer.zookeeper.connect=localhost:2181/
systems.kafka.producer.metadata.broker.list=localhost:9092
Note that, in this case, the system is called "kafka". In your example,
you'd use the "systems.appname.*" prefix (or change your test to call
getKafkaSystemConsumerConfig("kafka") ).
Kafka does come with a way to do this in their TestUtils class:
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/u
tils/TestUtils.scala?source=c
The method you're probably looking for is:
createConsumerProperties
And:
getProducerConfig
What I'd recommend is just creating a java.util.Properties instance, and
adding all appropriate configs to it (either with putAll, or put). Then
I'd use the MapConfig class to convert the Properties to a Config object:
val config = new MapConfig(myProps);
This is a bit easier than going the factory route that you're on right
now, and it can all be done inside your unit test, instead of having extra
properties files.
Cheers,
Chris
On 11/18/13 4:53 PM, "Garry Turkington" <g....@improvedigital.com>
wrote:
>Hi,
>
>I thought SAMZA-77 looked like fun so made the changes required to add
>the stringified UUID to the client and group IDs. All nice and easy.
>Then I tried adding some test cases and think I'm managing to make this
>more difficult than it needs to be. :)
>
>In my test case I have the following, which I believe is the right chain
>of calls to create a KafkaConfig object (I created a simple config file
>in the resources directory):
>
> val factory = new PropertiesConfigFactory()
> val config =
>factory.getConfig(URI.create("file://%s/src/test/resources/test.properties
><file:///\\%25s\src\test\resources\test.properties>" format new File
>(".").getCanonicalPath))
>val kafkaConfig = new KafkaConfig(config)
>
>All well and good but then I try and call the appropriate method on the
>KafkaConfig object (with some uncertainty of what the argument needs map
>to):
>
>val config1 = kafkaConfig.getKafkaSystemConsumerConfig("appname")
>
>This causes an exception to be thrown, the highlights being:
>
>java.lang.IllegalArgumentException: requirement failed: Missing required
>property 'zookeeper.connect'
> at scala.Predef$.require(Predef.scala:145)
> at
>kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:175)
> at kafka.utils.ZKConfig.<init>(ZkUtils.scala:775)
> at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:73)
> at kafka.consumer.ConsumerConfig.<init>(ConsumerConfig.scala:77)
> at
>org.apache.samza.config.KafkaConfig.getKafkaSystemConsumerConfig(KafkaConf
>ig.scala:91)
>
>This suggests that to create a ConsumerConfig requires the whole ZK/Kafka
>stack to be instantiated - is this the case and if so pointers to the
>minimum config set required to get it working? I tried to piece
>something together from the other test cases but couldn't find the right
>magic.
>
>Or is there an easier way to do all this? :)
>
>Thanks
>Garry