You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/04/09 07:42:16 UTC
[GitHub] [skywalking] EvanLjp opened a new issue #6719: Add lAL in Satellite
EvanLjp opened a new issue #6719:
URL: https://github.com/apache/skywalking/issues/6719
I want to convert log to metrics in Satellite to reuse the alarm plugin in the OAP server.
There are some ideas and problems about the future.
# Ideas
1.the counter which contains some special logs.
# problems
1. Do we need to be as complicated as LAL of the server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817849307
In general, the LAL design architecture in Java is referable in this issue, not all the features need to be implemented in this repo though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-816631619
Do you have any proposal about how to do log analysis in golang? Any design?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817846098
> I would implement 2 filters, which are log abort filter and log metrics Filter.
> So it's simpler than the LAL in the server.
The feature parity between Go version and Java version is not a big concern to me, what I'm concerned is to design the architecture carefully and to make it easily extend when we need to implement all the features in Java version, see my comments 👇.
According to your comment https://github.com/apache/skywalking/issues/6719#issuecomment-817803089 , I'm afraid that the implementation will be very specific to what you need, not in a general manner.
Because there is DSL in the Java version, which makes the LAL in Java looks more "modern", while I didn't find DSL in Go, here is my suggestions.
Because all discussed above are `filter` plugin, I'd suggest defining a general filter and some strategies (that can be defined by users and can be reused) to perform the actions after the filtered logs, here is a possible configuration.
```yaml
filters:
- regex: ".*DEBUG.*"
# ... other configurations
actions:
- abort
- regex: ".*failed to order a product.*"
# ... other configurations
actions:
- countErrorMetrics
- abort
- regex: ".*failed to order a product.*"
actions:
- label1
filterActions:
- countErrorMetrics:
name: "errorLogCount"
type: counter
# ... other
- abort: {}
- label1:
key: "{{ Log.Key }}"
val: "constantValue"
- label2:
key: "level"
val: "{{ Log.Level }}"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng edited a comment on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
wu-sheng edited a comment on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-912201608
@EvanLjp Are you still going to this? There are months for no update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] EvanLjp edited a comment on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
EvanLjp edited a comment on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817803089
Maybe there are 3 new plugins for the feature, which are log_abort_filter plugin, log_meter_filter_plugin, and the log_meter_tag plugin.
# log_abort_filter plugin
```
processor:
filters:
- plugin_name: "log-abort-filter"
aborts:
- service: demo_service
service_instance: demo_instance
endpoint: endpoint1
abort_tags:
- logger: logger1
- thread: thread1
```
# log_meter_filter_plugin
```
processor:
filters:
- plugin_name: "log-metrics-counter-filter"
log_metrics_rules_file: log-metrics-counter.file
[
{
"metrics_name1_counter":"",
"filter":[
{
"type":"regexp",
"service":"service1",
"serviceInstance":"instance1",
"endpoint":"endpoint1",
"tags":[
{
"tagname1":"tagvalue1"
}
],
"message":"message1"
}
]
}
]
```
# log_meter_tag plugin
```
[
{
"metrics_name1_counter": "",
"constant_labels": {
"a": "b"
},
"dynamic_labels": [
"service",
"serviceInstance",
"endpoint",
"tags.xx"
]
}
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] EvanLjp commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
EvanLjp commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-912202790
Sorry, I am worried about I don't have enough time to do the case. Maybe we can close it first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] EvanLjp edited a comment on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
EvanLjp edited a comment on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817558275
I would implement 2 filters, which are log abort filter and log metrics Filter.
So it's simpler than the LAL in the server.
The log abort filter would abort the log according to the tags or message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] EvanLjp edited a comment on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
EvanLjp edited a comment on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817558275
I would implement 2 filters, which are log abort filter and log metrics Filter.
So it's simpler than the LAL in the server.
The log abort filter would abort the log according to the tags or message.
And the log Metrics counter Filter would extract Metric by the tags, and the regular expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817875526
Let me take a look again at the current filter plugin implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #6719:
URL: https://github.com/apache/skywalking/issues/6719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] EvanLjp commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
EvanLjp commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817862512
but I don't want to make filter plugins more complex, I prefer to use the current filter chain to process the problem. Maybe our configuration could be more comfortable. The satellite has multi-pipes to process different telemetry data. The log pipe is only one of them. So I'm cautious about plug-in sharing configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] EvanLjp commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
EvanLjp commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817558275
I would implement 2 filters, which are log abort filter and log metrics Filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817811705
Where do the regex matching and splitting happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] EvanLjp commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
EvanLjp commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817803089
Maybe there are 3 new plugins for the feature, which are log_abort_filter plugin, log_meter_filter_plugin, and the log_meter_tag plugin.
# log_abort_filter plugin
```
processor:
filters:
- plugin_name: "log-abort-filter"
aborts:
- service: demo_service
service_instance: demo_instance
endpoint: endpoint1
abort_tags:
- logger: logger1
- thread: thread1
```
# log_meter_filter_plugin
```
processor:
filters:
- plugin_name: "log-metrics-counter-filter"
log_metrics_rules_file: log-metrics-counter.file
[
{
"metrics_name1_counter":"",
"filter":[
{
"type":"regexp",
"service":"service1",
"serviceInstance":"instance1",
"endpoint":"endpoint1",
"tags":[
{
"tagname1":"tagvalue1"
}
],
"message":"message1"
}
]
}
]
# log_meter_tag plugin
[
{
"metrics_name1_counter": "",
"constant_labels": {
"a": "b"
},
"dynamic_labels": [
"service",
"serviceInstance",
"endpoint",
"tags.xx"
]
}
]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-817800603
We should have abort/regex/re-tag/metric or more filters for go-version LAL. @kezhenxu94 Idea or suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6719: Add LAL in Satellite
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6719:
URL: https://github.com/apache/skywalking/issues/6719#issuecomment-912201608
@EvanLjp Are you still going to this? There are months for no udpate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org