You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/04/11 16:02:44 UTC
[GitHub] [skywalking] hailin0 commented on issue #6261: Support elastic / beats log-receiver
hailin0 commented on issue #6261:
URL: https://github.com/apache/skywalking/issues/6261#issuecomment-817330492
# demo
### logback pattern
```text
[%tc] [%level] %date{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %logger{96}:%line - %msg%n
```
### logback output to file
```text
[TC:N/A] [INFO] 2021-04-11 17:23:33.371 [thread-main] com.example.Main:100 - log message 1
[TC:[service-a,3a5b8da5a5ba40c0b192e91b5c80f1a8@192.168.1.8,N/A,-1,N/A]] [INFO] 2021-04-11 17:23:33.371 [thread-web-1] com.example.controller.TestController:100 - log message 2
[TC:[service-a,3a5b8da5a5ba40c0b192e91b5c80f1a8@192.168.1.8,ddd92f52207c468e9cd03ddd107cd530.69.16181331190470001,0,ddd92f52207c468e9cd03ddd107cd530.69.16181331190470000]] [INFO] 2021-04-11 17:25:19.062 [thread-web-2] com.example.controller.TestController:150 - log message 3
```
### filebeat.yml config
```yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /tmp/filebeat/filebeat-in.log
processors:
- dissect:
tokenizer: '[TC:%{TC}] [%{logTags.level}] %{datetime} [%{logTags.thread}] %{logTags.logger}:%{line} - %{body}'
field: "message"
target_prefix: ""
overwrite_keys: true
- dissect:
tokenizer: '[%{service},%{serviceInstance},%{traceContext.traceId},%{traceContext.spanId},%{traceContext.traceSegmentId}]'
field: "TC"
target_prefix: ""
- script:
lang: javascript
id: filter_datetime_format
source: >
function process(event) {
var datetime = event.Get("datetime");
var datetimeArr = datetime.split(" ");
var dateArr = datetimeArr[0].split("-");
var timeArrs = datetimeArr[1].split(".");
var timeArr = timeArrs[0].split(":");
var mills = timeArrs[1];
var date = new Date();
date.setUTCFullYear(parseInt(dateArr[0]));
date.setUTCMonth(parseInt(dateArr[1]) - 1);
date.setUTCDate(parseInt(dateArr[2]));
date.setUTCHours(parseInt(timeArr[0]));
date.setUTCMinutes(parseInt(timeArr[1]));
date.setUTCSeconds(parseInt(timeArr[2]));
date.setUTCMilliseconds(parseInt(mills));
var timeMillis = date.getTime();
event.Put("timestamp", timeMillis);
}
- copy_fields:
fields:
- from: log.file.path
to: logTags.logfile
- from: agent.hostname
to: logTags.hostname
- from: agent.type
to: logTags.agent
- from: agent.version
to: logTags.agentVersion
- from: agent.id
to: logTags.agentId
- from: agent.name
to: logTags.agentName
fail_on_error: false
ignore_missing: true
- drop_fields:
fields: ["@timestamp", "@metadata", "ecs", "host", "input", "agent", "log", "TC", "line", "message", "datetime"]
output.file:
path: "/tmp/filebeat"
filename: filebeat-out.log
```
### filebeat output json
```json
{
"@timestamp":"2021-04-11T15:40:49.025Z",
"@metadata":{
"beat":"filebeat",
"type":"_doc",
"version":"7.12.0"
},
"service":"service-a",
"serviceInstance":"3a5b8da5a5ba40c0b192e91b5c80f1a8@192.168.1.8",
"traceContext":{
"traceId":"ddd92f52207c468e9cd03ddd107cd530.69.16181331190470001",
"spanId":"0",
"traceSegmentId":"ddd92f52207c468e9cd03ddd107cd530.69.16181331190470000"
},
"logTags":{
"thread":"thread-web-1",
"level":"INFO",
"logger":"com.example.controller.TestController",
"hostname":"5d960e25ed4c",
"filepath":"/tmp/filebeat/filebeat-in.log",
"agent":"filebeat",
"agentId":"9744570a-0e17-43c8-8958-c5f102ef7eb5",
"agentName":"5d960e25ed4c",
"agentVersion":"7.12.0"
},
"timestamp":1618161813371,
"body":"log message 3"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org