You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/11/05 16:08:24 UTC
[GitHub] [skywalking] BFergerson opened a new issue #8083: [Feature] query log count
BFergerson opened a new issue #8083:
URL: https://github.com/apache/skywalking/issues/8083
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
Currently, you can query logs via `ILogQueryDAO` but you cannot just query log counts. I think we should add a parameter that indicates that only `Logs.total` should be populated. That way you can quickly get a feel for how many logs there are without having to fetch them all.
### Use case
I just want to know how many of a particular format of a log pattern was captured at a particular time. I don't want the log data.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] BFergerson edited a comment on issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
BFergerson edited a comment on issue #8083:
URL: https://github.com/apache/skywalking/issues/8083#issuecomment-962469630
@wu-sheng, yeah you're right. I'm currently learning more about LAL/MAL and now that you mention it, that's obviously how I should do it. I won't be able to get counts for logs that happened prior to adding the new meters, but that'll be fine. Moving forward they'll have queriable counts. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #8083:
URL: https://github.com/apache/skywalking/issues/8083#issuecomment-962379427
> I just want to know how many of a particular format of a log pattern was captured at a particular time. I don't want the log data.
I think we need to be more clear about what is a particular format? Could you provide some examples?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] BFergerson commented on issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
BFergerson commented on issue #8083:
URL: https://github.com/apache/skywalking/issues/8083#issuecomment-962469630
@wu-sheng, yeah you're right. I'm currently learning more about LAL/MAL and now that you mention it that's obviously how I should do it. I won't be able to get counts for logs that happened prior to adding the new meters, but that'll be fine. Moving forward they'll have queriable counts. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 edited a comment on issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
kezhenxu94 edited a comment on issue #8083:
URL: https://github.com/apache/skywalking/issues/8083#issuecomment-962296521
Storage like h2 MySQL don't support pattern searching, and storage like ElasticSearch doesn't return the exact total count of result. Is that what you can tolerate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #8083:
URL: https://github.com/apache/skywalking/issues/8083#issuecomment-962468699
> I use plugin.toolkit.log.transmit_formatted=false so I don't care so much about the log data. I just care about the tags and the number of times each pattern has occurred. So if I can get the count, I can calculate log density.
Have you taken a look at how LAL(Log Analysis Language) engine works? It could fetch the tags(regex) and generate metrics for MAL(metrics analysis language script) engine to get all the number related things you want. In your case, I think it is `sum`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] BFergerson commented on issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
BFergerson commented on issue #8083:
URL: https://github.com/apache/skywalking/issues/8083#issuecomment-962467327
@kezhenxu94, that actually does look like it'll work for H2. Maybe not so much for ES. Either way, I'll close this issue. I was trying to come up with a way to query log counts without writing H2/ES specific code, but I think the custom code is necessary.
@wu-sheng, I use `plugin.toolkit.log.transmit_formatted=false` so I don't care so much about the log data. I just care about the tags and the number of times each pattern has occurred. So if I can get the count, I can calculate log density.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] BFergerson closed issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
BFergerson closed issue #8083:
URL: https://github.com/apache/skywalking/issues/8083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #8083:
URL: https://github.com/apache/skywalking/issues/8083#issuecomment-962296521
Storage like h2 MySQL don't support pattern searching, and storage like ElasticSearch doesn't return the exact total count of result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 edited a comment on issue #8083: [Feature] query log count
Posted by GitBox <gi...@apache.org>.
kezhenxu94 edited a comment on issue #8083:
URL: https://github.com/apache/skywalking/issues/8083#issuecomment-962296521
> Currently, you can query logs via `ILogQueryDAO` but you cannot just query log counts.
Does passing `limit = 0` do the trick? I think it is.
Also, storage like h2 MySQL don't support pattern searching, and storage like ElasticSearch doesn't return the exact total count of result. Is that what you can tolerate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org