You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2019/12/19 04:14:54 UTC
[GitHub] [skywalking] adamni135 opened a new issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
adamni135 opened a new issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
when i use kafka producer.send(new ProducerRecord<String, byte[]>(topic, msg), callback);
callback must not be null
we found lots of threads waiting at org.apache.kafka.clients.producer.internals.BufferPool#allocate line134
codes: waitingTimeElapsed = !moreMemory.await(remainingTimeToBlockNs, TimeUnit.NANOSECONDS);
when we delete org.apache.skywalking.apm.plugin.kafka.CallbackInterceptor#beforeMethod
line43-48 then our springboot application performance will be as usual
___
### Bug
- Which version of SkyWalking, OS and JRE?
kafka maven:
org.apache.kafka kafka_2.12 2.2.0
skywalking plugin:
apm-kafka-plugin-6.5.0.jar
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
when using kafka sender with callback will degrade performance very much
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] adamni135 opened a new issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
adamni135 opened a new issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
- [x] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
when i use kafka producer.send(new ProducerRecord<String, byte[]>(topic, msg), callback);
callback must not be null
we found lots of threads waiting at org.apache.kafka.clients.producer.internals.BufferPool#allocate line134
codes: waitingTimeElapsed = !moreMemory.await(remainingTimeToBlockNs, TimeUnit.NANOSECONDS);
when we delete org.apache.skywalking.apm.plugin.kafka.CallbackInterceptor#beforeMethod
line43-48 then our springboot application performance will be as usual
___
### Bug
- Which version of SkyWalking, OS and JRE?
kafka maven:
org.apache.kafka kafka_2.12 2.2.0
skywalking plugin:
apm-kafka-plugin-6.5.0.jar
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
when using kafka sender with callback will degrade performance very much
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087#issuecomment-567962233
You could find the QQ group here, https://github.com/apache/skywalking#contact-us. Basically, tech discussion should be on the mail list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087#issuecomment-567913439
We usually don't output the error unless it is necessary. Your suggestion is hard to implement. This is already logging API, how to log when it is already blocked :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] adamni135 closed issue #4087: when i use skywalking,
i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
adamni135 closed issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] adamni135 commented on issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
adamni135 commented on issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087#issuecomment-567914827
I know it is a part of log, so may be it can log it self as next record, when we check the log we can know the queue is full, so that we can raise the log level or any other things, and this is only for suggestion,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] adamni135 commented on issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
adamni135 commented on issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087#issuecomment-567906800
we found the root cause ,my team mate use debug level log, we change it to error level then it fixed, sorry to trouble you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] adamni135 commented on issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
adamni135 commented on issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087#issuecomment-567895069
maybe we found something new
this lock cause by the deallocate buffer delay too much
we found the code in kafka how to deallocate the buffer is:
org.apache.kafka.clients.producer.internals.Sender.completeBatch(ProducerBatch, PartitionResponse)
if (batch.done(response.baseOffset, response.logAppendTime, null)) { // here kafka will call the callback
maybeRemoveFromInflightBatches(batch);
this.accumulator.deallocate(batch); // here kafka will release the buffer
}
so if the callback rt is too large, it will cause the buffer can not release as soon as possible
and we trace in this method and print the stack:
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:82)
at com.bigo.eco.common.kafka.sender.KafkaEmptyCallback.onCompletion(KafkaEmptyCallback.java:-1)
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion$original$qobEMB62(KafkaProducer.java:1304)
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion$original$qobEMB62$accessor$KxcVlwF4(KafkaProducer.java:-1)
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback$auxiliary$SW3k8KDv.call(null:-1)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:93)
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java:-1)
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:227)
at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:196)
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:677)
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:649)
at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:557)
at org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74)
at org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:786)
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:557)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
at java.lang.Thread.run(Thread.java:748)
and the method
org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(long, long, RuntimeException) use 10ms to run
it is too long for the buffer to reuse
so we trace inside of the method
and found the each "thunk.callback.onCompletion(metadata, null);" cause 2ms
and we need to find why it case 2ms
then we get this trace
`---+Tracing for : thread_name="kafka-producer-network-thread | producer-1" thread_id=0x39;is_daemon=true;priority=5;
`---+[2,2ms]org.apache.skywalking.apm.agent.core.context.TracingContext:finish()
+---[0,0ms]java.util.LinkedList:isEmpty(@452)
+---[0,0ms]org.apache.skywalking.apm.agent.core.context.TracingContext:isLimitMechanismWorking(@453)
+---[0,0ms]org.apache.skywalking.apm.agent.core.context.trace.TraceSegment:finish(@453)
+---[0,0ms]org.apache.skywalking.apm.agent.core.context.trace.TraceSegment:hasRef(@460)
+---[0,0ms]org.apache.skywalking.apm.agent.core.context.trace.TraceSegment:createTime(@471)
`---[2,2ms]org.apache.skywalking.apm.agent.core.context.TracingContext$ListenerManager:notifyFinish(@475)
the code:
org.apache.skywalking.apm.agent.core.context.TracingContext$ListenerManager:notifyFinish(@475)
used 2ms
could you let me know what's this method for ? and do you have some suggest to fix it？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] adamni135 commented on issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
adamni135 commented on issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087#issuecomment-567912243
some suggestion, the FileWriter write method:
/**
* Write log to the queue. W/ performance trade off, set 2ms timeout for the log OP.
*
* @param message to log
*/
@Override public void write(String message) {
try {
logBuffer.offer(message, 2, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
the offer will return true or false,
maybe you can log when it return false;
that the developer can more easier to figger out the thread has been blocked
best wishes!!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087#issuecomment-567915097
I know your meaning, but it is hard to implement. If you have any idea to do this in a good way, a pull request is welcome.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] adamni135 closed issue #4087: when i use skywalking,
i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
adamni135 closed issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] adamni135 commented on issue #4087: when i use
skywalking, i founded kafkaProcuctor will degrade performance very much
Posted by GitBox <gi...@apache.org>.
adamni135 commented on issue #4087: when i use skywalking, i founded kafkaProcuctor will degrade performance very much
URL: https://github.com/apache/skywalking/issues/4087#issuecomment-567951858
ok，，let me think think how to fix it , do you have wechat ? can i join your chat group？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services