[GitHub] [skywalking] hailin0 edited a comment on issue #6261: Support elastic / beats log-receiver

[GitBox <gi...@apache.org>]

hailin0 edited a comment on issue #6261:
URL: https://github.com/apache/skywalking/issues/6261#issuecomment-817330492


   # logback demo
   
   ### logback pattern
   ```text
   [%tc] [%level] %date{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %logger{96}:%line - %msg%n
   ```
   
   ### logback output to file
   ```text
   [TC:N/A] [INFO] 2021-04-11 17:23:33.371 [thread-main] com.example.Main:100 - log message 1
   [TC:[service-a,3a5b8da5a5ba40c0b192e91b5c80f1a8@192.168.1.8,N/A,-1,N/A]] [INFO] 2021-04-11 17:23:33.371 [thread-web-1] com.example.controller.TestController:100 - log message 2
   [TC:[service-a,3a5b8da5a5ba40c0b192e91b5c80f1a8@192.168.1.8,ddd92f52207c468e9cd03ddd107cd530.69.16181331190470001,0,ddd92f52207c468e9cd03ddd107cd530.69.16181331190470000]] [INFO] 2021-04-11 17:25:19.062 [thread-web-2] com.example.controller.TestController:150 - log message 3
   ```
   
   ### filebeat.yml config
   ```yml
   filebeat.inputs:
   - type: log
     enabled: true
     paths:
       - /tmp/filebeat/filebeat-in.log
   
   processors:
     - dissect:
         tokenizer: '[TC:%{TC}] [%{logTags.level}] %{datetime} [%{logTags.thread}] %{logTags.logger}:%{line} - %{body}'
         field: "message"
         target_prefix: ""
         overwrite_keys: true
     - dissect:
         tokenizer: '[%{service},%{serviceInstance},%{traceContext.traceId},%{traceContext.spanId},%{traceContext.traceSegmentId}]'
         field: "TC"
         target_prefix: ""
     - script:
         lang: javascript
         id: filter_datetime_format
         source: >
           function process(event) {
               var datetime = event.Get("datetime");
               var datetimeArr = datetime.split(" ");
               var dateArr = datetimeArr[0].split("-");
               var timeArrs = datetimeArr[1].split(".");
               var timeArr = timeArrs[0].split(":");
               var mills = timeArrs[1];
   
               var date = new Date();
               date.setUTCFullYear(parseInt(dateArr[0]));
               date.setUTCMonth(parseInt(dateArr[1]) - 1);
               date.setUTCDate(parseInt(dateArr[2]));
               date.setUTCHours(parseInt(timeArr[0]));
               date.setUTCMinutes(parseInt(timeArr[1]));
               date.setUTCSeconds(parseInt(timeArr[2]));
               date.setUTCMilliseconds(parseInt(mills));
   
               var timeMillis = date.getTime();
               event.Put("timestamp", timeMillis);
           }
     - copy_fields:
         fields:
           - from: log.file.path
             to: logTags.logfile
           - from: agent.hostname
             to: logTags.hostname
           - from: agent.type
             to: logTags.agent
           - from: agent.version
             to: logTags.agentVersion
           - from: agent.id
             to: logTags.agentId
           - from: agent.name
             to: logTags.agentName
         fail_on_error: false
         ignore_missing: true
     - drop_fields:
         fields: ["@timestamp", "@metadata", "ecs", "host", "input", "agent", "log", "TC", "line", "message", "datetime"]
   
   output.file:
     path: "/tmp/filebeat"
     filename: filebeat-out.log
   ```
   
   
   ### filebeat output json to kafka
   ```json
   {
       "@timestamp":"2021-04-11T15:40:49.025Z",
       "@metadata":{
           "beat":"filebeat",
           "type":"_doc",
           "version":"7.12.0"
       },
       "service":"service-a",
       "serviceInstance":"3a5b8da5a5ba40c0b192e91b5c80f1a8@192.168.1.8",
       "traceContext":{
           "traceId":"ddd92f52207c468e9cd03ddd107cd530.69.16181331190470001",
           "spanId":"0",
           "traceSegmentId":"ddd92f52207c468e9cd03ddd107cd530.69.16181331190470000"
       },
       "logTags":{
           "thread":"thread-web-1",
           "level":"INFO",
           "logger":"com.example.controller.TestController",
           "hostname":"5d960e25ed4c",
           "filepath":"/tmp/filebeat/filebeat-in.log",
           "agent":"filebeat",
           "agentId":"9744570a-0e17-43c8-8958-c5f102ef7eb5",
           "agentName":"5d960e25ed4c",
           "agentVersion":"7.12.0"
       },
       "timestamp":1618161813371,
       "body":"log message 3"
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org