Embedded pulsar cluster?

[jun aoki <ja...@apache.org>]

do you guys have embedded pulsar I can use for unit tests?
Something similar embedded kafka to
https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/utils/EmbeddedKafkaCluster.java

I'd like to play with pulsar by replacing our kafka stream app with it.

-- 
-jun

Re: Embedded pulsar cluster?

[jun aoki <ja...@apache.org>]

Matteo, It was nice meeting your earlier and thank you for letting me those
options.
I am new so I can't picture what form of embedded/single class mini-pulsar
cluster would be ideal for our unit tests yet, so I'd have to try them and
get a feel of it first. I'm also looking into pulsar docker image and it
seems I can quickly make up some local integration with our services. (
https://hub.docker.com/r/apachepulsar/pulsar/tags/)

On Fri, May 4, 2018 at 4:50 PM, Matteo Merli <ma...@gmail.com> wrote:

> Hi Jun,
>
> there are few options:
>
>  1. We have a MockedBrokerService that we use in our unit tests. It's not
> super-fancy but it might work:
> https://github.com/apache/incubator-pulsar/blob/master/
> pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/
> MockedPulsarServiceBaseTest.java
>    That starts a "real" broker instance with mocked ZooKeeper and
> BookKeeper to keep data in memory. The way we typically use this is by
> extending from that class and create tests.
>
> 2. There is a standalone Pulsar service ("bin/pulsar standalone") that
> includes broker, ZK and BK service in a single JVM process. It uses the
> "real" implementation of each component and stores the data on disk. It was
> not initially meant to be embedded but it might actually very easy to fix.
>
> 3. There's a LocalBookkeeperEnsemble class that creates a ZK + BK service
> and, combined with a PulsarService instance, could get the whole service
> up. I agree that a single class "service" would be a nicer option.
>
> Matteo
>
> On Fri, May 4, 2018 at 12:55 PM jun aoki <ja...@apache.org> wrote:
>
> > do you guys have embedded pulsar I can use for unit tests?
> > Something similar embedded kafka to
> >
> > https://github.com/apache/kafka/blob/trunk/streams/src/
> test/java/org/apache/kafka/streams/integration/utils/
> EmbeddedKafkaCluster.java
> >
> > I'd like to play with pulsar by replacing our kafka stream app with it.
> >
> > --
> > -jun
> >
> --
> Matteo Merli
> <mm...@apache.org>
>



-- 
-jun

Re: Embedded pulsar cluster?

[Matteo Merli <ma...@gmail.com>]

Hi Jun,

there are few options:

 1. We have a MockedBrokerService that we use in our unit tests. It's not
super-fancy but it might work:
https://github.com/apache/incubator-pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/MockedPulsarServiceBaseTest.java
   That starts a "real" broker instance with mocked ZooKeeper and
BookKeeper to keep data in memory. The way we typically use this is by
extending from that class and create tests.

2. There is a standalone Pulsar service ("bin/pulsar standalone") that
includes broker, ZK and BK service in a single JVM process. It uses the
"real" implementation of each component and stores the data on disk. It was
not initially meant to be embedded but it might actually very easy to fix.

3. There's a LocalBookkeeperEnsemble class that creates a ZK + BK service
and, combined with a PulsarService instance, could get the whole service
up. I agree that a single class "service" would be a nicer option.

Matteo

On Fri, May 4, 2018 at 12:55 PM jun aoki <ja...@apache.org> wrote:

> do you guys have embedded pulsar I can use for unit tests?
> Something similar embedded kafka to
>
> https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/utils/EmbeddedKafkaCluster.java
>
> I'd like to play with pulsar by replacing our kafka stream app with it.
>
> --
> -jun
>
-- 
Matteo Merli
<mm...@apache.org>