Interest in using ActiveMQ as a trace data transport for Zipkin?

[Adrian Cole <ad...@apache.org>]

Hi, all.

The Apache Zipkin(incubating) project allows distributed tracing by sending trace identifiers in application payloads usually as headers, with timing data sent out of band often by a messaging transport like RabbitMQ and Kafka. We have some community interest in supporting ActiveMQ also, including an implementation. This would mean that apps would implicitly send trace data over an ActiveMQ connection and that Zipkin server would poll that on the other side. This is particularly a big decision for us as we are considering ActiveMQ as an out-of-box feature.

As Zipkin is for distributed tracing, that means potentially many languages, at least java, go, ruby, php, python etc. The libraries needed to support this are not a lot of code, but something we'd want to maintain. I know there are a couple ways to use ActiveMQ, and possibly some constraints or version concerns of interest. If anyone has feedback on using ActiveMQ to move distributed trace data, please add your points of interest here. This could be of advice nature, or that you would use it, or otherwise

issue: https://github.com/openzipkin/zipkin/issues/1990
work in progress server integration: https://github.com/openzipkin/zipkin/pull/2466
work in progress java-tracer integration: https://github.com/openzipkin/zipkin-reporter-java/pull/136

Re: Interest in using ActiveMQ as a trace data transport for Zipkin?

[Tim Bain <tb...@alumni.duke.edu>]

The STOMP protocol and the AMQP 1.0 protocol are the most language-portable
protocols with which you can interact with ActiveMQ. If you were able to
limit yourselves to integrating with Java apps, JMS message brokers become
an option, but your description doesn't sound like that's where you're at.
So I'd suggest you use STOMP or AMQP 1.0. Please note that AMQP 1.0 is not
compatible with older versions of the AMQP protocol (0-9-1 and earlier), so
pay attention to the version number that client libraries support if you go
this route.

ActiveMQ 5.x, ActiveMQ Artemis, and RabbitMQ all support both of those
protocols (for RabbitMQ, it's apparently via a plugin since the native
broker implementation of AMQP is 0-9-1), so with either of those protocols
you should be able to use the same client library no matter which of those
three brokers you connect to. Kafka doesn't support either of those
protocols (nor anything else that's compatible with another messaging
middleware product, so far as I know), so you'll need a way to configure
your clients to use its client libraries when that's the broker product.

Justin's questions about things like how long the data is supposed to be
retained and whether a user expects to be able to do retroactive
consumption after the data is sent are good ones for determining if there's
a fit for either of the flavors of ActiveMQ. For example, both brokers (and
all JMS servers, to my knowledge, plus RabbitMQ) discard messages after
they are consumed by all consumers, whereas Kafka, Kinesis, Pulsar,
RocketMQ, etc. retain their distributed journals until the data ages off.
Depending on your needs, either or both paradigms might be acceptable, but
without knowing your specific needs it's hard to say whether either flavor
of ActiveMQ would meet your needs. If both do, it should be simple to use
the same STOMP or AMQP 1.0 client library to connect to either one (or
RabbitMQ), so you wouldn't need to actually make a choice between them;
that choice could be left up to your users.

Tim

On Mon, Mar 25, 2019 at 8:13 PM Justin Bertram <jb...@apache.org> wrote:

> Can you explain a bit more the model used for sending/consuming the trace
> data and how long does the data is supposed to stick around? Does it follow
> a pub/sub model? If the data sticks around awhile then a traditional
> message broker might not be the greatest option as messages are meant to be
> consumed pretty quickly rather than accumulating in a queue. Once messages
> start accumulating you start running into performance degradation due to
> paging.
>
> As far as I know RabbitMQ doesn't have its own messaging transport (unlike
> Kakfa). It was built around AMQP so if the RabbitMQ integration is using
> AMQP 1.0 then it should be pretty simple to also integrate with ActiveMQ
> 5.x and/or ActiveMQ Artemis or any other broker that supports AMQP 1.0.
>
> It appears from the discussions on GitHub that the 5.x broker is being
> targeted. However, for what it's worth, ActiveMQ Artemis is slated to
> become ActiveMQ 6 once it reaches sufficient feature parity.
>
>
> Justin
>
> On Mon, Mar 25, 2019 at 7:27 PM Adrian Cole <ad...@apache.org> wrote:
>
> > Hi, all.
> >
> > The Apache Zipkin(incubating) project allows distributed tracing by
> > sending trace identifiers in application payloads usually as headers,
> with
> > timing data sent out of band often by a messaging transport like RabbitMQ
> > and Kafka. We have some community interest in supporting ActiveMQ also,
> > including an implementation. This would mean that apps would implicitly
> > send trace data over an ActiveMQ connection and that Zipkin server would
> > poll that on the other side. This is particularly a big decision for us
> as
> > we are considering ActiveMQ as an out-of-box feature.
> >
> > As Zipkin is for distributed tracing, that means potentially many
> > languages, at least java, go, ruby, php, python etc. The libraries needed
> > to support this are not a lot of code, but something we'd want to
> maintain.
> > I know there are a couple ways to use ActiveMQ, and possibly some
> > constraints or version concerns of interest. If anyone has feedback on
> > using ActiveMQ to move distributed trace data, please add your points of
> > interest here. This could be of advice nature, or that you would use it,
> or
> > otherwise
> >
> > issue: https://github.com/openzipkin/zipkin/issues/1990
> > work in progress server integration:
> > https://github.com/openzipkin/zipkin/pull/2466
> > work in progress java-tracer integration:
> > https://github.com/openzipkin/zipkin-reporter-java/pull/136
> >
>

Re: Interest in using ActiveMQ as a trace data transport for Zipkin?

[Justin Bertram <jb...@apache.org>]

Can you explain a bit more the model used for sending/consuming the trace
data and how long does the data is supposed to stick around? Does it follow
a pub/sub model? If the data sticks around awhile then a traditional
message broker might not be the greatest option as messages are meant to be
consumed pretty quickly rather than accumulating in a queue. Once messages
start accumulating you start running into performance degradation due to
paging.

As far as I know RabbitMQ doesn't have its own messaging transport (unlike
Kakfa). It was built around AMQP so if the RabbitMQ integration is using
AMQP 1.0 then it should be pretty simple to also integrate with ActiveMQ
5.x and/or ActiveMQ Artemis or any other broker that supports AMQP 1.0.

It appears from the discussions on GitHub that the 5.x broker is being
targeted. However, for what it's worth, ActiveMQ Artemis is slated to
become ActiveMQ 6 once it reaches sufficient feature parity.


Justin

On Mon, Mar 25, 2019 at 7:27 PM Adrian Cole <ad...@apache.org> wrote:

> Hi, all.
>
> The Apache Zipkin(incubating) project allows distributed tracing by
> sending trace identifiers in application payloads usually as headers, with
> timing data sent out of band often by a messaging transport like RabbitMQ
> and Kafka. We have some community interest in supporting ActiveMQ also,
> including an implementation. This would mean that apps would implicitly
> send trace data over an ActiveMQ connection and that Zipkin server would
> poll that on the other side. This is particularly a big decision for us as
> we are considering ActiveMQ as an out-of-box feature.
>
> As Zipkin is for distributed tracing, that means potentially many
> languages, at least java, go, ruby, php, python etc. The libraries needed
> to support this are not a lot of code, but something we'd want to maintain.
> I know there are a couple ways to use ActiveMQ, and possibly some
> constraints or version concerns of interest. If anyone has feedback on
> using ActiveMQ to move distributed trace data, please add your points of
> interest here. This could be of advice nature, or that you would use it, or
> otherwise
>
> issue: https://github.com/openzipkin/zipkin/issues/1990
> work in progress server integration:
> https://github.com/openzipkin/zipkin/pull/2466
> work in progress java-tracer integration:
> https://github.com/openzipkin/zipkin-reporter-java/pull/136
>