You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Gerald Nunn <ge...@gmail.com> on 2020/05/03 05:53:22 UTC
Opentracing and Kafka Propagation?
I'm using camel 2.23 with Spring Boot for a set of microservices that send
and receive messages via Kafka. I have the individual services working with
Jaegar with no issue however I was expecting transactions to span Kafka in
Jaeger and I'm not seeing that. Rather in Jaeger the transactions stop and
start with Kafka.
In my pom.xml file (
https://github.com/gnunn1/seating/blob/master/registration-ui/pom.xml) I'm
importing the following dependencies for tracing:
<!-- Tracing -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-opentracing-starter</artifactId>
</dependency>
<dependency>
<groupId>io.jaegertracing</groupId>
<artifactId>jaeger-client</artifactId>
<version>0.32.0</version>
</dependency>
<dependency>
<groupId>io.jaegertracing</groupId>
<artifactId>jaeger-tracerresolver</artifactId>
<version>0.32.0</version>
</dependency>
</dependencies>
My application has the @CamelOpenTracing annotation on the Application
class. Finally I'm connecting to Kafka in Spring DSL as follows:
<route id="request-seat" routePolicyRef="policy">
...
<to id="send-seat" uri="kafka:registration?clientId=registration-ui"/>
</route>
My understanding is that camel should propagate the tracing context
automatically but maybe I'm missing something? I see there is a
KafkaSpanDecorator in camel but it's not clear if this is an internal class
or something I should be using explicitly.
Thanks,
Gerald
Re: Opentracing and Kafka Propagation?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Thanks for reporting back here too. Yeah this would be great to get
resolved so we can have nice tracing from kafka.
Maybe Gary can help you with providing a fix for us - he has developed
the opentracing component AFAIR
On Wed, May 6, 2020 at 7:46 PM Gerald Nunn <ge...@gmail.com> wrote:
>
> So thanks to some help from Gary Brown we realized the issue is that
> headers in Kafka are byte arrays and not strings. I've opened a JIRA issue
> for it here:
>
> https://issues.apache.org/jira/browse/CAMEL-15022
>
> Cheers,
>
> Gerald
>
> On Sun, 3 May 2020 at 15:06, Gerald Nunn <ge...@gmail.com> wrote:
>
> > So I am still stuck on this but some additional information. I enabled
> > trace logging in camel-opentracing and it looks like the appropriate code
> > is being hit. Unfortunately KafkaSpanDecorator doesn't have any trace or
> > debug logs so I do not see anything from that.
> >
> > Producer Logging:
> >
> > 21:33:40.575 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> > 21:33:40.582 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start server
> > span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> > 21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start server
> > span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> > 21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start server
> > span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> > 21:33:40.587 [XNIO-1 task-1] INFO request-seat - Starting route
> > 21:33:40.601 [XNIO-1 task-1] INFO request-seat -
> > com.redhat.demo.Registration@27bbd37
> > 21:33:40.647 [XNIO-1 task-1] INFO request-seat -
> > {"candidatename":"test","email":"test@test.com
> > ","seatname":"724","seatno":"19_22"}
> > 21:33:40.648 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start client
> > span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
> > 21:33:40.706 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start client
> > span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
> > 21:33:40.708 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] INFO i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 -
> > registration
> > 21:33:40.710 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: finish server
> > span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] INFO i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: finish server
> > span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] INFO i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: finish server
> > span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] INFO i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> > 21:33:40.712 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> > 21:33:40.712 [XNIO-1 task-1] INFO i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> >
> > Consumer Logging:
> >
> > 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> > span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> > 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> > span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> > 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> > span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> > 21:33:40.723 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> > reserve-seat - >>> {"candidatename":"test","email":"test@test.com
> > ","seatname":"724","seatno":"19_22"}
> > 21:33:40.729 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> > reserve-seat - >>> {"candidatename":"test","email":"test@test.com
> > ","seatname":"724","seatno":"19_22"}
> > 21:33:40.730 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
> > span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
> > span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> > i.j.i.reporters.LoggingReporter - Span reported:
> > bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> > span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> > i.j.i.reporters.LoggingReporter - Span reported:
> > bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> > span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> > i.j.i.reporters.LoggingReporter - Span reported:
> > 1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> > span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> > i.j.i.reporters.LoggingReporter - Span reported:
> > 72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> >
> > On the Jaeger fight it shows the following detals for the producer in the
> > trace when you drill in:
> >
> >
> > camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
> > component: "camel-kafka"
> > internal.span.format: "jaeger"
> > message_bus.destination: "registration"
> > span.kind: "producer"
> >
> > And on the consumer side I see:
> >
> >
> > camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
> > component: "camel-kafka"
> > internal.span.format: "jaeger"
> > kafka.offset: "909"
> > kafka.partition: "0"
> > message_bus.destination: "registration"
> > sampler.param: true
> > sampler.type: "const"
> > span.kind: "consumer"
> >
> > Any suggestions on how to debug this would be appreciated.
> >
> > Cheers,
> >
> > Gerald
> >
> > On Sat, 2 May 2020 at 22:53, Gerald Nunn <ge...@gmail.com> wrote:
> >
> >> I'm using camel 2.23 with Spring Boot for a set of microservices that
> >> send and receive messages via Kafka. I have the individual services working
> >> with Jaegar with no issue however I was expecting transactions to span
> >> Kafka in Jaeger and I'm not seeing that. Rather in Jaeger the transactions
> >> stop and start with Kafka.
> >>
> >> In my pom.xml file (
> >> https://github.com/gnunn1/seating/blob/master/registration-ui/pom.xml)
> >> I'm importing the following dependencies for tracing:
> >>
> >> <!-- Tracing -->
> >> <dependency>
> >> <groupId>org.apache.camel</groupId>
> >> <artifactId>camel-opentracing-starter</artifactId>
> >> </dependency>
> >> <dependency>
> >> <groupId>io.jaegertracing</groupId>
> >> <artifactId>jaeger-client</artifactId>
> >> <version>0.32.0</version>
> >> </dependency>
> >> <dependency>
> >> <groupId>io.jaegertracing</groupId>
> >> <artifactId>jaeger-tracerresolver</artifactId>
> >> <version>0.32.0</version>
> >> </dependency>
> >> </dependencies>
> >>
> >> My application has the @CamelOpenTracing annotation on the Application
> >> class. Finally I'm connecting to Kafka in Spring DSL as follows:
> >>
> >> <route id="request-seat" routePolicyRef="policy">
> >> ...
> >> <to id="send-seat" uri="kafka:registration?clientId=registration-ui"/>
> >> </route>
> >>
> >> My understanding is that camel should propagate the tracing context
> >> automatically but maybe I'm missing something? I see there is a
> >> KafkaSpanDecorator in camel but it's not clear if this is an internal class
> >> or something I should be using explicitly.
> >>
> >> Thanks,
> >>
> >> Gerald
> >>
> >
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Opentracing and Kafka Propagation?
Posted by Gerald Nunn <ge...@gmail.com>.
So thanks to some help from Gary Brown we realized the issue is that
headers in Kafka are byte arrays and not strings. I've opened a JIRA issue
for it here:
https://issues.apache.org/jira/browse/CAMEL-15022
Cheers,
Gerald
On Sun, 3 May 2020 at 15:06, Gerald Nunn <ge...@gmail.com> wrote:
> So I am still stuck on this but some additional information. I enabled
> trace logging in camel-opentracing and it looks like the appropriate code
> is being hit. Unfortunately KafkaSpanDecorator doesn't have any trace or
> debug logs so I do not see anything from that.
>
> Producer Logging:
>
> 21:33:40.575 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> 21:33:40.582 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start server
> span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> 21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start server
> span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> 21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start server
> span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> 21:33:40.587 [XNIO-1 task-1] INFO request-seat - Starting route
> 21:33:40.601 [XNIO-1 task-1] INFO request-seat -
> com.redhat.demo.Registration@27bbd37
> 21:33:40.647 [XNIO-1 task-1] INFO request-seat -
> {"candidatename":"test","email":"test@test.com
> ","seatname":"724","seatno":"19_22"}
> 21:33:40.648 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start client
> span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
> 21:33:40.706 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start client
> span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
> 21:33:40.708 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] INFO i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 -
> registration
> 21:33:40.710 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: finish server
> span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] INFO i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: finish server
> span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] INFO i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: finish server
> span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] INFO i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> 21:33:40.712 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> 21:33:40.712 [XNIO-1 task-1] INFO i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
>
> Consumer Logging:
>
> 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> 21:33:40.723 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> reserve-seat - >>> {"candidatename":"test","email":"test@test.com
> ","seatname":"724","seatno":"19_22"}
> 21:33:40.729 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> reserve-seat - >>> {"candidatename":"test","email":"test@test.com
> ","seatname":"724","seatno":"19_22"}
> 21:33:40.730 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
> span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
> span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> i.j.i.reporters.LoggingReporter - Span reported:
> bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> i.j.i.reporters.LoggingReporter - Span reported:
> bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> i.j.i.reporters.LoggingReporter - Span reported:
> 1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> i.j.i.reporters.LoggingReporter - Span reported:
> 72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
>
> On the Jaeger fight it shows the following detals for the producer in the
> trace when you drill in:
>
>
> camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
> component: "camel-kafka"
> internal.span.format: "jaeger"
> message_bus.destination: "registration"
> span.kind: "producer"
>
> And on the consumer side I see:
>
>
> camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
> component: "camel-kafka"
> internal.span.format: "jaeger"
> kafka.offset: "909"
> kafka.partition: "0"
> message_bus.destination: "registration"
> sampler.param: true
> sampler.type: "const"
> span.kind: "consumer"
>
> Any suggestions on how to debug this would be appreciated.
>
> Cheers,
>
> Gerald
>
> On Sat, 2 May 2020 at 22:53, Gerald Nunn <ge...@gmail.com> wrote:
>
>> I'm using camel 2.23 with Spring Boot for a set of microservices that
>> send and receive messages via Kafka. I have the individual services working
>> with Jaegar with no issue however I was expecting transactions to span
>> Kafka in Jaeger and I'm not seeing that. Rather in Jaeger the transactions
>> stop and start with Kafka.
>>
>> In my pom.xml file (
>> https://github.com/gnunn1/seating/blob/master/registration-ui/pom.xml)
>> I'm importing the following dependencies for tracing:
>>
>> <!-- Tracing -->
>> <dependency>
>> <groupId>org.apache.camel</groupId>
>> <artifactId>camel-opentracing-starter</artifactId>
>> </dependency>
>> <dependency>
>> <groupId>io.jaegertracing</groupId>
>> <artifactId>jaeger-client</artifactId>
>> <version>0.32.0</version>
>> </dependency>
>> <dependency>
>> <groupId>io.jaegertracing</groupId>
>> <artifactId>jaeger-tracerresolver</artifactId>
>> <version>0.32.0</version>
>> </dependency>
>> </dependencies>
>>
>> My application has the @CamelOpenTracing annotation on the Application
>> class. Finally I'm connecting to Kafka in Spring DSL as follows:
>>
>> <route id="request-seat" routePolicyRef="policy">
>> ...
>> <to id="send-seat" uri="kafka:registration?clientId=registration-ui"/>
>> </route>
>>
>> My understanding is that camel should propagate the tracing context
>> automatically but maybe I'm missing something? I see there is a
>> KafkaSpanDecorator in camel but it's not clear if this is an internal class
>> or something I should be using explicitly.
>>
>> Thanks,
>>
>> Gerald
>>
>
Re: Opentracing and Kafka Propagation?
Posted by Gerald Nunn <ge...@gmail.com>.
So I am still stuck on this but some additional information. I enabled
trace logging in camel-opentracing and it looks like the appropriate code
is being hit. Unfortunately KafkaSpanDecorator doesn't have any trace or
debug logs so I do not see anything from that.
Producer Logging:
21:33:40.575 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
21:33:40.582 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start server
span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start server
span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start server
span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
21:33:40.587 [XNIO-1 task-1] INFO request-seat - Starting route
21:33:40.601 [XNIO-1 task-1] INFO request-seat -
com.redhat.demo.Registration@27bbd37
21:33:40.647 [XNIO-1 task-1] INFO request-seat -
{"candidatename":"test","email":"test@test.com
","seatname":"724","seatno":"19_22"}
21:33:40.648 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start client
span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
21:33:40.706 [Camel (Registration) thread #1 - KafkaProducer[registration]]
TRACE o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
21:33:40.708 [Camel (Registration) thread #1 - KafkaProducer[registration]]
INFO i.j.i.reporters.LoggingReporter - Span reported:
ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
21:33:40.710 [Camel (Registration) thread #1 - KafkaProducer[registration]]
TRACE o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
INFO i.j.i.reporters.LoggingReporter - Span reported:
ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
TRACE o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
INFO i.j.i.reporters.LoggingReporter - Span reported:
ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
TRACE o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
INFO i.j.i.reporters.LoggingReporter - Span reported:
ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
21:33:40.712 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
21:33:40.712 [XNIO-1 task-1] INFO i.j.i.reporters.LoggingReporter - Span
reported: ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
Consumer Logging:
21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
21:33:40.723 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
reserve-seat - >>> {"candidatename":"test","email":"test@test.com
","seatname":"724","seatno":"19_22"}
21:33:40.729 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
reserve-seat - >>> {"candidatename":"test","email":"test@test.com
","seatname":"724","seatno":"19_22"}
21:33:40.730 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
i.j.i.reporters.LoggingReporter - Span reported:
bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
i.j.i.reporters.LoggingReporter - Span reported:
bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
i.j.i.reporters.LoggingReporter - Span reported:
1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
i.j.i.reporters.LoggingReporter - Span reported:
72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
On the Jaeger fight it shows the following detals for the producer in the
trace when you drill in:
camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
component: "camel-kafka"
internal.span.format: "jaeger"
message_bus.destination: "registration"
span.kind: "producer"
And on the consumer side I see:
camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
component: "camel-kafka"
internal.span.format: "jaeger"
kafka.offset: "909"
kafka.partition: "0"
message_bus.destination: "registration"
sampler.param: true
sampler.type: "const"
span.kind: "consumer"
Any suggestions on how to debug this would be appreciated.
Cheers,
Gerald
On Sat, 2 May 2020 at 22:53, Gerald Nunn <ge...@gmail.com> wrote:
> I'm using camel 2.23 with Spring Boot for a set of microservices that send
> and receive messages via Kafka. I have the individual services working with
> Jaegar with no issue however I was expecting transactions to span Kafka in
> Jaeger and I'm not seeing that. Rather in Jaeger the transactions stop and
> start with Kafka.
>
> In my pom.xml file (
> https://github.com/gnunn1/seating/blob/master/registration-ui/pom.xml)
> I'm importing the following dependencies for tracing:
>
> <!-- Tracing -->
> <dependency>
> <groupId>org.apache.camel</groupId>
> <artifactId>camel-opentracing-starter</artifactId>
> </dependency>
> <dependency>
> <groupId>io.jaegertracing</groupId>
> <artifactId>jaeger-client</artifactId>
> <version>0.32.0</version>
> </dependency>
> <dependency>
> <groupId>io.jaegertracing</groupId>
> <artifactId>jaeger-tracerresolver</artifactId>
> <version>0.32.0</version>
> </dependency>
> </dependencies>
>
> My application has the @CamelOpenTracing annotation on the Application
> class. Finally I'm connecting to Kafka in Spring DSL as follows:
>
> <route id="request-seat" routePolicyRef="policy">
> ...
> <to id="send-seat" uri="kafka:registration?clientId=registration-ui"/>
> </route>
>
> My understanding is that camel should propagate the tracing context
> automatically but maybe I'm missing something? I see there is a
> KafkaSpanDecorator in camel but it's not clear if this is an internal class
> or something I should be using explicitly.
>
> Thanks,
>
> Gerald
>