You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2021/09/26 11:44:54 UTC
[GitHub] [incubator-inlong] cxntsh opened a new issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
cxntsh opened a new issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592
<!--
Thanks for contributing to InLong!
Please explain your issue precisely, and if possible provide a reproducer
snippet (this helps resolve issues much quicker).
-->
#### Expected behavior
#### Actual behavior
When using TextFileReader to read the log file, the CPU usage rate is very high, nearly 50%. After the file is read, there is also a usage rate of nearly 20%.
#### Steps to reproduce
I took a look at the source code, the while loop inside, there is empty polling for reading, which takes up too much CPU clock, can it be optimized?
@Override
public Message read() {
if (iterator != null && iterator.hasNext()) {
String message = iterator.next();
if (validateMessage(message)) {
textFileMetric.readNum.incr();
return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8));
}
}
AgentUtils.silenceSleepInMs(100); // This seems to work, is there a better way ?
return null;
}
#### System configuration
**InLong version**: x.y
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] cxntsh commented on issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
cxntsh commented on issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592#issuecomment-927291632
`
public
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] ruanwenjun commented on issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592#issuecomment-939264606
Does this caused by the loop in `TaskWrapper`?
https://github.com/apache/incubator-inlong/blob/73d149610e1d88913a65529a5bb003bf1ea058a7/inlong-agent/agent-core/src/main/java/org/apache/inlong/agent/core/task/TaskWrapper.java#L80-L94
Currently, the `isFinished` will wait until timeout, can we just return true, if the `iterator.hasNext() == false`, and add sleep if we set `NEVER_STOP_SIGN`.
https://github.com/apache/incubator-inlong/blob/73d149610e1d88913a65529a5bb003bf1ea058a7/inlong-agent/agent-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/TextFileReader.java#L95-L111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] cxntsh commented on issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
cxntsh commented on issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592#issuecomment-927291533
I took a look at the source code, the while loop inside, there is empty polling for reading, which takes up too much CPU clock, can it be optimized?
I try this code:
`@Override
public Message read() {
if (iterator != null && iterator.hasNext()) {
String message = iterator.next();
if (validateMessage(message)) {
textFileMetric.readNum.incr();
return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8));
}
}
AgentUtils.silenceSleepInMs(100); // This seems to work, is there a better way ??
return null;
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] EMsnap commented on issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
EMsnap commented on issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592#issuecomment-930281175
good point @dockerzhang pls check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] cxntsh removed a comment on issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
cxntsh removed a comment on issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592#issuecomment-927291632
`
public
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] cxntsh removed a comment on issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
cxntsh removed a comment on issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592#issuecomment-927291533
I took a look at the source code, the while loop inside, there is empty polling for reading, which takes up too much CPU clock, can it be optimized?
I try this code:
`@Override
public Message read() {
if (iterator != null && iterator.hasNext()) {
String message = iterator.next();
if (validateMessage(message)) {
textFileMetric.readNum.incr();
return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8));
}
}
AgentUtils.silenceSleepInMs(100); // This seems to work, is there a better way ??
return null;
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] cxntsh commented on issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
cxntsh commented on issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592#issuecomment-927291867
I took a look at the source code, the while loop inside, there is empty polling for reading, which takes up too much CPU clock, can it be optimized?
```
@Override
public Message read() {
if (iterator != null && iterator.hasNext()) {
String message = iterator.next();
if (validateMessage(message)) {
textFileMetric.readNum.incr();
return new DefaultMessage(message.getBytes(StandardCharsets.UTF_8));
}
}
AgentUtils.silenceSleepInMs(100); // This seems to work, is there a better way ??
return null;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] EMsnap commented on issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
EMsnap commented on issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592#issuecomment-930281175
good point @dockerzhang pls check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-inlong] aloyszhang closed issue #1592: TextFileReader: The cpu utilization rate is very high, nearly 50%
Posted by GitBox <gi...@apache.org>.
aloyszhang closed issue #1592:
URL: https://github.com/apache/incubator-inlong/issues/1592
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org