Kafka Connect startup issues

[Jeff Klukas <jk...@simple.com>]

I'm doing some local testing on my Mac to get a feel for Kafka Connect, and
I'm running into several issues.

First, when I untar the Kafka 0.10.0.1 source and run
`./bin/connect-distributed.sh config/connect-distributed.properties`, I get
a "Usage" message. By digging through scripts a bit, I found that the
CLASSPATH variable ends up being empty, so it's interpreting the properties
file as the classpath and thus thinks no properties file was passed.

In order to get past that hurdle, I had to find the compiled
`connect-runtime` jar and specifically set the classpath to look at that
jar. Is it expected that the connect-distributed.sh script should be usable
directly from the Kafka source? Am I missing documentation about setup?

Second, I'm finding that I can't get the Kafka Connect process to shut down
cleanly. The DistributedHerder always hangs on shutdown. When I hit Ctrl-C,
I see the following log messages:

INFO  [2016-09-10 01:11:40,430]
org.apache.kafka.connect.runtime.Connect: Kafka Connect stopping
INFO  [2016-09-10 01:11:40,430]
org.apache.kafka.connect.runtime.rest.RestServer: Stopping REST server
INFO  [2016-09-10 01:11:40,443]
org.apache.kafka.connect.runtime.rest.RestServer: REST server stopped
INFO  [2016-09-10 01:11:40,443]
org.apache.kafka.connect.runtime.distributed.DistributedHerder: Herder
stopping

But the "Herder stopped" message never comes and I can only stop the
process by calling `kill -9` on it. Any thoughts on what could cause this?

Re: Kafka Connect startup issues

[Shikhar Bhushan <sh...@confluent.io>]

Hi Jeff,

If you are using the source tar, you will first need to compile the JAR's
as per the instructions on the README (https://github.com/apache/kafka).
Then you will indeed be able to run with "./bin/connect-distributed.sh
config/connect-distributed.properties". Alternately, you can just use the
binary distribution.

For the shutdown issue, I have opened KAFKA-4154
<https://issues.apache.org/jira/browse/KAFKA-4154>. I was able to reproduce
when starting Connect without Kafka running, but it shutdown correctly when
Kafka was running as it was able to complete initialization.

Best,

Shikhar

On Mon, Sep 12, 2016 at 5:54 AM Jeff Klukas <jk...@simple.com> wrote:

> I'm doing some local testing on my Mac to get a feel for Kafka Connect, and
> I'm running into several issues.
>
> First, when I untar the Kafka 0.10.0.1 source and run
> `./bin/connect-distributed.sh config/connect-distributed.properties`, I get
> a "Usage" message. By digging through scripts a bit, I found that the
> CLASSPATH variable ends up being empty, so it's interpreting the properties
> file as the classpath and thus thinks no properties file was passed.
>
> In order to get past that hurdle, I had to find the compiled
> `connect-runtime` jar and specifically set the classpath to look at that
> jar. Is it expected that the connect-distributed.sh script should be usable
> directly from the Kafka source? Am I missing documentation about setup?
>
> Second, I'm finding that I can't get the Kafka Connect process to shut down
> cleanly. The DistributedHerder always hangs on shutdown. When I hit Ctrl-C,
> I see the following log messages:
>
> INFO  [2016-09-10 01:11:40,430]
> org.apache.kafka.connect.runtime.Connect: Kafka Connect stopping
> INFO  [2016-09-10 01:11:40,430]
> org.apache.kafka.connect.runtime.rest.RestServer: Stopping REST server
> INFO  [2016-09-10 01:11:40,443]
> org.apache.kafka.connect.runtime.rest.RestServer: REST server stopped
> INFO  [2016-09-10 01:11:40,443]
> org.apache.kafka.connect.runtime.distributed.DistributedHerder: Herder
> stopping
>
> But the "Herder stopped" message never comes and I can only stop the
> process by calling `kill -9` on it. Any thoughts on what could cause this?
>