You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Chinmay Kolhatkar <ch...@apache.org> on 2016/10/13 14:25:58 UTC

EmbeddedKafka Server for testing kafka related code

Dear Community,

While testing kafka related endpoint code for Apex-Calcite integration, I
found an interesting embedded kafka implementation over the internet and I
used it in my code.

The implementation is based on ready to use EmbeddedZookeer and KafkaServer
from mvn dependency kafka_2.11:0.9.0.0:test.

I found this local kafka server very stable for testing purpose. Would it
be a good addition to malhar-kafka module in test folder?

The EmbeddedKafka server is here:
https://github.com/chinmaykolhatkar/apex-malhar/blob/calcite/sql/src/test/java/org/apache/apex/malhar/sql/EmbeddedKafka.java

In above class, I've also added method for publishing and consume from
topic.

Please share your opinion.

Thanks,
Chinmay.

Re: EmbeddedKafka Server for testing kafka related code

Posted by Chinmay Kolhatkar <ch...@apache.org>.
I can look into possibility of using this in existing tests as well.

Basically the idea behind this EmbeddedKafka class is to provide a single
object which spawns up JVM-local Kafka Cluster.

Possibly there are extensions (like multiple brokers etc) which needs to be
done in this class for making use in existing Kafka test. I'll investigate
that.






On Thu, Oct 13, 2016 at 11:27 PM, Thomas Weise <th...@gmail.com>
wrote:

> +1, how hard is it to change the existing tests in the Kafka module to use
> it?
>
> Different versions are IMO not an issue for the existing tests, they
> already run against a fixed version (the broker is launched as part of the
> test).
>
> On Thu, Oct 13, 2016 at 9:59 AM, Sanjay Pujare <sa...@datatorrent.com>
> wrote:
>
> > Strong +1 for using embedded kafka. Found embedded servers very useful in
> > ActiveMQ and SQS (ElasticMQ) unit tests. My only question/concern is how
> > well can you simulate different versions of Kafka.
> >
> > On 10/13/16, 7:25 AM, "Chinmay Kolhatkar" <ch...@apache.org> wrote:
> >
> >     Dear Community,
> >
> >     While testing kafka related endpoint code for Apex-Calcite
> > integration, I
> >     found an interesting embedded kafka implementation over the internet
> > and I
> >     used it in my code.
> >
> >     The implementation is based on ready to use EmbeddedZookeer and
> > KafkaServer
> >     from mvn dependency kafka_2.11:0.9.0.0:test.
> >
> >     I found this local kafka server very stable for testing purpose.
> Would
> > it
> >     be a good addition to malhar-kafka module in test folder?
> >
> >     The EmbeddedKafka server is here:
> >     https://github.com/chinmaykolhatkar/apex-malhar/
> > blob/calcite/sql/src/test/java/org/apache/apex/malhar/
> > sql/EmbeddedKafka.java
> >
> >     In above class, I've also added method for publishing and consume
> from
> >     topic.
> >
> >     Please share your opinion.
> >
> >     Thanks,
> >     Chinmay.
> >
> >
> >
> >
>

Re: EmbeddedKafka Server for testing kafka related code

Posted by Thomas Weise <th...@gmail.com>.
+1, how hard is it to change the existing tests in the Kafka module to use
it?

Different versions are IMO not an issue for the existing tests, they
already run against a fixed version (the broker is launched as part of the
test).

On Thu, Oct 13, 2016 at 9:59 AM, Sanjay Pujare <sa...@datatorrent.com>
wrote:

> Strong +1 for using embedded kafka. Found embedded servers very useful in
> ActiveMQ and SQS (ElasticMQ) unit tests. My only question/concern is how
> well can you simulate different versions of Kafka.
>
> On 10/13/16, 7:25 AM, "Chinmay Kolhatkar" <ch...@apache.org> wrote:
>
>     Dear Community,
>
>     While testing kafka related endpoint code for Apex-Calcite
> integration, I
>     found an interesting embedded kafka implementation over the internet
> and I
>     used it in my code.
>
>     The implementation is based on ready to use EmbeddedZookeer and
> KafkaServer
>     from mvn dependency kafka_2.11:0.9.0.0:test.
>
>     I found this local kafka server very stable for testing purpose. Would
> it
>     be a good addition to malhar-kafka module in test folder?
>
>     The EmbeddedKafka server is here:
>     https://github.com/chinmaykolhatkar/apex-malhar/
> blob/calcite/sql/src/test/java/org/apache/apex/malhar/
> sql/EmbeddedKafka.java
>
>     In above class, I've also added method for publishing and consume from
>     topic.
>
>     Please share your opinion.
>
>     Thanks,
>     Chinmay.
>
>
>
>

Re: EmbeddedKafka Server for testing kafka related code

Posted by Sanjay Pujare <sa...@datatorrent.com>.
Strong +1 for using embedded kafka. Found embedded servers very useful in ActiveMQ and SQS (ElasticMQ) unit tests. My only question/concern is how well can you simulate different versions of Kafka.

On 10/13/16, 7:25 AM, "Chinmay Kolhatkar" <ch...@apache.org> wrote:

    Dear Community,
    
    While testing kafka related endpoint code for Apex-Calcite integration, I
    found an interesting embedded kafka implementation over the internet and I
    used it in my code.
    
    The implementation is based on ready to use EmbeddedZookeer and KafkaServer
    from mvn dependency kafka_2.11:0.9.0.0:test.
    
    I found this local kafka server very stable for testing purpose. Would it
    be a good addition to malhar-kafka module in test folder?
    
    The EmbeddedKafka server is here:
    https://github.com/chinmaykolhatkar/apex-malhar/blob/calcite/sql/src/test/java/org/apache/apex/malhar/sql/EmbeddedKafka.java
    
    In above class, I've also added method for publishing and consume from
    topic.
    
    Please share your opinion.
    
    Thanks,
    Chinmay.