You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Stefan Kölbl <ko...@gmx.at> on 2017/02/15 15:27:49 UTC

Cannot start up Kafka Server within Java

Dear Kafka dev team!

 

I’m implementing a data stream processor with Apache Kafka for my bachelor’s
thesis and right now I’m struggling with trying to get the Kafka Server
started with Java.

 

A short summary of what I’m doing currently:

1.	Start ZooKeeper (manually via terminal).
2.	Start Kafka Server (manually via terminal).
3.	Create Kafka Topic (manually via terminal, although it would be
auto-created by the producer anyway).
4.	Run Kafka Producer written with the Kafka Clients API in Java.
5.	Run Kafka Consumer written with the Kafka Clients API in Java.

 

To further automatize the testing process with jUnit, I’d like to be able to
start ZooKeeper and the Kafka Server with Java too, as I do with the
producer and the consumer.

This should be possible, according to some examples I found online, using
the Scala source of Kafka (not Kafka Clients!):
http://www.programcreek.com/java-api-examples/index.php?api=kafka.server.Kaf
kaServer

But whenever I try to create a new KafkaServer object, I get an error
(please see attached error.png).

 

Here are my maven dependencies:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.10.1.1</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
    <version>0.10.1.1</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.11</artifactId>
    <version>0.10.1.1</version>
</dependency>

 

I Somehow have the feeling I’m missing something crucial and easy-to-fix.
Neither a lot of google searches nor playing around with the parameters
(time, threadNamePrefix, kafkaMetricsReporters) provided to the KafkaServer
constructor could resolve my issue.

Could you please help me? I’m stuck and don’t know what to do anymore.

Thank you in advance.

 

Best regards,

Stefan Kölbl

Re: Cannot start up Kafka Server within Java

Posted by radai <ra...@gmail.com>.
kafka publishes a "test" artifact:
http://search.maven.org/#search|ga|1|a%3A%22kafka_2.12%22
if you introduce a test-scoped dep. on it, which in maven would look like:
<dependency>
   <groupId>org.apache.kafka
<http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.kafka%22>
</groupId>
   <artifactId>kafka_2.12</artifactId>
   <version>0.10.1.1</version>
   <classifier>test</classifier>
   <scope>test</scope>
</dependency>
you would get access to trait KafkaServerTestHarness, that you can base
your unit tests on (warning - its scala)

On Wed, Feb 15, 2017 at 8:33 AM, Mathieu Fenniak <
mathieu.fenniak@replicon.com> wrote:

> Hi Stefan,
>
> The mailing list suppressed your attachment, so, it's hard to offer any
> advice without the error.  I'd suggest trying to post the error in a GitHub
> Gist, or some similar format.
>
> I dug up this mailing list archive about starting a Kafka server in-process
> that might help:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201608.mbox/%
> 3CCAJTjOQG5MdboQ9ciBrvZNLQzaDAvKhyQeW-mmz-br4bWy32B7A%40mail.gmail.com%3E
>  The whole thread (Automated Testing w/ Kafka Streams) has a few other
> approaches as well that you might be interested in trying.
>
> Mathieu
>
>
> On Wed, Feb 15, 2017 at 8:27 AM, Stefan Kölbl <ko...@gmx.at> wrote:
>
> > Dear Kafka dev team!
> >
> >
> >
> > I’m implementing a data stream processor with Apache Kafka for my
> > bachelor’s thesis and right now I’m struggling with trying to get the
> Kafka
> > Server started with Java.
> >
> >
> >
> > A short summary of what I’m doing currently:
> >
> >    1. Start ZooKeeper (manually via terminal).
> >    2. Start Kafka Server (manually via terminal).
> >    3. Create Kafka Topic (manually via terminal, although it would be
> >    auto-created by the producer anyway).
> >    4. Run Kafka Producer written with the Kafka Clients API in Java.
> >    5. Run Kafka Consumer written with the Kafka Clients API in Java.
> >
> >
> >
> > To further automatize the testing process with jUnit, I’d like to be able
> > to start ZooKeeper and the Kafka Server with Java too, as I do with the
> > producer and the consumer.
> >
> > This should be possible, according to some examples I found online, using
> > the Scala source of Kafka (not Kafka Clients!):
> > http://www.programcreek.com/java-api-examples/index.php?
> > api=kafka.server.KafkaServer
> >
> > But whenever I try to create a new KafkaServer object, I get an error
> > (please see attached error.png).
> >
> >
> >
> > Here are my maven dependencies:
> >
> > <*dependency*>
> >     <*groupId*>org.apache.kafka</*groupId*>
> >     <*artifactId*>kafka-clients</*artifactId*>
> >     <*version*>0.10.1.1</*version*>
> > </*dependency*>
> > <*dependency*>
> >     <*groupId*>org.apache.kafka</*groupId*>
> >     <*artifactId*>kafka-streams</*artifactId*>
> >     <*version*>0.10.1.1</*version*>
> > </*dependency*>
> > <*dependency*>
> >     <*groupId*>org.apache.kafka</*groupId*>
> >     <*artifactId*>kafka_2.11</*artifactId*>
> >     <*version*>0.10.1.1</*version*>
> > </*dependency*>
> >
> >
> >
> > I Somehow have the feeling I’m missing something crucial and easy-to-fix.
> > Neither a lot of google searches nor playing around with the parameters
> > (time, threadNamePrefix, kafkaMetricsReporters) provided to the
> KafkaServer
> > constructor could resolve my issue.
> >
> > Could you please help me? I’m stuck and don’t know what to do anymore.
> >
> > Thank you in advance.
> >
> >
> >
> > Best regards,
> >
> > Stefan Kölbl
> >
>

Re: Cannot start up Kafka Server within Java

Posted by Mathieu Fenniak <ma...@replicon.com>.
Hi Stefan,

The mailing list suppressed your attachment, so, it's hard to offer any
advice without the error.  I'd suggest trying to post the error in a GitHub
Gist, or some similar format.

I dug up this mailing list archive about starting a Kafka server in-process
that might help:
http://mail-archives.apache.org/mod_mbox/kafka-users/201608.mbox/%3CCAJTjOQG5MdboQ9ciBrvZNLQzaDAvKhyQeW-mmz-br4bWy32B7A%40mail.gmail.com%3E
 The whole thread (Automated Testing w/ Kafka Streams) has a few other
approaches as well that you might be interested in trying.

Mathieu


On Wed, Feb 15, 2017 at 8:27 AM, Stefan Kölbl <ko...@gmx.at> wrote:

> Dear Kafka dev team!
>
>
>
> I’m implementing a data stream processor with Apache Kafka for my
> bachelor’s thesis and right now I’m struggling with trying to get the Kafka
> Server started with Java.
>
>
>
> A short summary of what I’m doing currently:
>
>    1. Start ZooKeeper (manually via terminal).
>    2. Start Kafka Server (manually via terminal).
>    3. Create Kafka Topic (manually via terminal, although it would be
>    auto-created by the producer anyway).
>    4. Run Kafka Producer written with the Kafka Clients API in Java.
>    5. Run Kafka Consumer written with the Kafka Clients API in Java.
>
>
>
> To further automatize the testing process with jUnit, I’d like to be able
> to start ZooKeeper and the Kafka Server with Java too, as I do with the
> producer and the consumer.
>
> This should be possible, according to some examples I found online, using
> the Scala source of Kafka (not Kafka Clients!):
> http://www.programcreek.com/java-api-examples/index.php?
> api=kafka.server.KafkaServer
>
> But whenever I try to create a new KafkaServer object, I get an error
> (please see attached error.png).
>
>
>
> Here are my maven dependencies:
>
> <*dependency*>
>     <*groupId*>org.apache.kafka</*groupId*>
>     <*artifactId*>kafka-clients</*artifactId*>
>     <*version*>0.10.1.1</*version*>
> </*dependency*>
> <*dependency*>
>     <*groupId*>org.apache.kafka</*groupId*>
>     <*artifactId*>kafka-streams</*artifactId*>
>     <*version*>0.10.1.1</*version*>
> </*dependency*>
> <*dependency*>
>     <*groupId*>org.apache.kafka</*groupId*>
>     <*artifactId*>kafka_2.11</*artifactId*>
>     <*version*>0.10.1.1</*version*>
> </*dependency*>
>
>
>
> I Somehow have the feeling I’m missing something crucial and easy-to-fix.
> Neither a lot of google searches nor playing around with the parameters
> (time, threadNamePrefix, kafkaMetricsReporters) provided to the KafkaServer
> constructor could resolve my issue.
>
> Could you please help me? I’m stuck and don’t know what to do anymore.
>
> Thank you in advance.
>
>
>
> Best regards,
>
> Stefan Kölbl
>