Bi-directional Bridge

[Michael André Pearce <mi...@me.com>]

Hi All,

We are looking at having two clusters in two countries where we need to bridge a queue named the same, from cluster a to cluster b and vice versa. Essentially achieving a bi directional eventual replicated cluster.

In kafka we are able to achieve this using a mirror maker - very akin to a bridge. And a custom handler which adds a header so that we can filter on this header to avoid a cycle in replication.

In artemis the custom handler equivalent seems to be the transformer, happy days :) and we have a configurable filter already too :)

The only small bit missing is being able to pass in custom config to the transformer so we can configure the transformer differently in each cluster.

I was thinking about extending the bridge config to have a sub section of config that is un-schema'd that gets passed into the transformer on creation.

Anyone any other ideas either this feature already exists or a better way to think about implementing this within artemis?

Cheers
Mike



Sent from my iPhone

Re: Bi-directional Bridge

[Michael André Pearce <mi...@me.com>]

I was wanting to make it configurable, rather than concrete class per cluster. Because we would have several pairs of clusters it's not just the one pairing. As such it makes more sense to be configurable, as like filter in the bridge already is today.

I like the idea of supporting key value pairs though. 

Cheers
Mike

Sent from my iPhone

> On 7 Sep 2017, at 23:48, Justin Bertram <jb...@redhat.com> wrote:
> 
> Could you just have a unique transformer in each cluster
> (com.foo.TransformerX, com.foo.TransformerY)?  That would be pretty
> straight-forward and would require no broker changes.
> 
> You could also use the same transformer class, but configure it with system
> properties.  That also would avoid changing the broker.
> 
> If you're set on adding a new feature for this why not have a set of
> key/value pairs that could be passed into the transformer?
> 
> 
> Justin
> 
> On Thu, Sep 7, 2017 at 5:39 PM, Michael André Pearce <
> michael.andre.pearce@me.com> wrote:
> 
>> Hi All,
>> 
>> We are looking at having two clusters in two countries where we need to
>> bridge a queue named the same, from cluster a to cluster b and vice versa.
>> Essentially achieving a bi directional eventual replicated cluster.
>> 
>> In kafka we are able to achieve this using a mirror maker - very akin to a
>> bridge. And a custom handler which adds a header so that we can filter on
>> this header to avoid a cycle in replication.
>> 
>> In artemis the custom handler equivalent seems to be the transformer,
>> happy days :) and we have a configurable filter already too :)
>> 
>> The only small bit missing is being able to pass in custom config to the
>> transformer so we can configure the transformer differently in each cluster.
>> 
>> I was thinking about extending the bridge config to have a sub section of
>> config that is un-schema'd that gets passed into the transformer on
>> creation.
>> 
>> Anyone any other ideas either this feature already exists or a better way
>> to think about implementing this within artemis?
>> 
>> Cheers
>> Mike
>> 
>> 
>> 
>> Sent from my iPhone

Re: Bi-directional Bridge

[Justin Bertram <jb...@redhat.com>]

Could you just have a unique transformer in each cluster
(com.foo.TransformerX, com.foo.TransformerY)?  That would be pretty
straight-forward and would require no broker changes.

You could also use the same transformer class, but configure it with system
properties.  That also would avoid changing the broker.

If you're set on adding a new feature for this why not have a set of
key/value pairs that could be passed into the transformer?


Justin

On Thu, Sep 7, 2017 at 5:39 PM, Michael André Pearce <
michael.andre.pearce@me.com> wrote:

> Hi All,
>
> We are looking at having two clusters in two countries where we need to
> bridge a queue named the same, from cluster a to cluster b and vice versa.
> Essentially achieving a bi directional eventual replicated cluster.
>
> In kafka we are able to achieve this using a mirror maker - very akin to a
> bridge. And a custom handler which adds a header so that we can filter on
> this header to avoid a cycle in replication.
>
> In artemis the custom handler equivalent seems to be the transformer,
> happy days :) and we have a configurable filter already too :)
>
> The only small bit missing is being able to pass in custom config to the
> transformer so we can configure the transformer differently in each cluster.
>
> I was thinking about extending the bridge config to have a sub section of
> config that is un-schema'd that gets passed into the transformer on
> creation.
>
> Anyone any other ideas either this feature already exists or a better way
> to think about implementing this within artemis?
>
> Cheers
> Mike
>
>
>
> Sent from my iPhone