[GitHub] [incubator-flagon-useralejs] poorejc commented on issue #243: sendLogs array includes nulls/missing entries

[GitBox <gi...@apache.org>]

poorejc commented on issue #243:
URL: https://github.com/apache/incubator-flagon-useralejs/issues/243#issuecomment-1107683022

   @EandrewJones Hi! Thanks for the ticket!
   
   A few more questions:
   
   1. You say that the nulls are "breaking" logStash Parsing. Can you clarify a bit what you mean there? Are you getting dropped logs? Or are you just getting errors like the legacy "grokparsefailure" from Logstash?
   
   2. What version of Logstash are you using? 6.8 or 7+ (note that we'll be moving to 7.8 soon, and have a working backend that will be moved over to Apache--this is a different build than the previous (@apache-flagon repo) build. See [here](https://github.com/UMD-ARLIS/docker-elk/blob/flagon-configs). 
   
   3. Are you using modified your own modified Logstash parsers/plugins?
   
   5. Which version of UserALE.js you are using? 2.2.0 or testing at 2.3.0? If 2.30, did you notice the same behavior in 2.2.0?
   
   6. Can you tell me a bit more about how you generated that first log. I'm guessing this is the view of the message from LogStash... There are some fields I'm not familiar with, e.g., `postIds`. `logType`=="raw", yet there are new fields added. Is this a custom log of sorts? or is there decoration through userale.map API? Any customer AuthHeaders added?
   
   Regarding `null` and LogStash... We are aware that fields with `null` values aren't parsed, particularly in fields like `location` and `details`. Examples (I used 2.2.0 to generate this, as the `toolVersion` in your logs is 2.2.0):
   
   Here is a `submit` log from the Example page from the example server (server.js):
   
   ```
     {
       target: 'form#test_text_input',
       path: [ 'form#test_text_input', 'body', 'html', '#document', 'Window' ],
       pageUrl: 'file:///
       pageTitle: 'UserALE.js - Example Page',
       pageReferrer: '',
       browser: { browser: 'chrome', version: '100.0.4896' },
       clientTime: 1650762410279,
       microTime: 0.3,
       location: { x: null, y: null },
       scrnRes: { width: 1280, height: 577 },
       type: 'submit',
       logType: 'raw',
       userAction: true,
       details: null,
       userId: 'me',
       toolVersion: '2.2.0',
       toolName: 'Apache UserALE.js Example (Custom)',
       useraleVersion: '2.2.0',
       sessionID: 'session_1650636958755'
     }
   ```
   `details` and `location` both include `null` values.
   
   Here is a `submit` log in Elasticsearch (in JSON):
   ```
   {
     "_index": "userale",
     "_type": "_doc",
     "_id": "vGAKWYABe4GJJBS6pZlw",
     "_version": 1,
     "_score": null,
     "fields": {
       "logType": [
         "raw"
       ],
       "userAction": [
         true
       ],
       "microTime": [
         0
       ],
       "pageTitle": [
         "UserALE.js - Example Page"
       ],
       "patch_ver": [
         0
       ],
       "browser.browser": [
         "chrome"
       ],
       "sessionID": [
         "session_1650636958755"
       ],
       "type": [
         "submit"
       ],
       "clientTime": [
         "2022-04-24T00:48:31.457Z"
       ],
       "scrnRes.width": [
         1280
       ],
       "userId": [
         "me"
       ],
       "target": [
         "form#test_text_input"
       ],
       "scrnRes.height": [
         577
       ],
       "minor_ver": [
         2
       ],
       "path": [
         "form#test_text_input",
         "body",
         "html",
         "#document",
         "Window"
       ],
       "toolVersion": [
         "2.2.0"
       ],
       "browser.version": [
         "100.0.4896"
       ],
       "@timestamp": [
         "2022-04-24T00:48:31.495Z"
       ],
       "useraleVersion": [
         "2.2.0"
       ],
       "host": [
         "172.20.0.1"
       ],
       "major_ver": [
         2
       ],
       "pageUrl": [
         "file:///C:/
       ],
       "pageReferrer": [
         ""
       ],
       "toolName": [
         "Apache UserALE.js Example (Custom)"
       ]
     },
     "highlight": {
       "type": [
         "@kibana-highlighted-field@submit@/kibana-highlighted-field@"
       ]
     },
     "sort": [
       1650761311457
     ]
   }
   ```
   Those fields are noticeably absent, although we're still getting `submit` logs.
   
   I generated these from the exact same test page... ELK is dropping dropping fields with null values (which is an indexing issue--the value null doesn't conform to our [index](https://github.com/UMD-ARLIS/docker-elk/blob/flagon-configs/logstash/templates/userale.json). So far, this has been fine b/c if its null its not really essential to have that data (although can create issues in edge cases for back end analyses).
   
   The null values I see in your `logs` output are as expected. Also, I think @UncleGedd is right about the empty lines after `submit` events--`submit` events trigger a page reload that is likely to blame for that empty line given how server.js works. However, I would like to figure out your first example, though. I've never seen null keys like that before in a UserALE log--very interested in how it was created.
   
   Happy to help out! Please answer those questions above as they'll be useful to us in doing forensics. 
   
   Thanks again @EandrewJones !
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@flagon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org