[GitHub] [metron] mmiklavc commented on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default

[GitBox <gi...@apache.org>]

mmiklavc commented on issue #1393: METRON-2092: Config UI does not require you to set a grok timestamp field by default
URL: https://github.com/apache/metron/pull/1393#issuecomment-498733822
 
 
   @sardell This comes from the grok pattern used. The value is an indirection to another variable that references the actual timestamp the user wants to use. For example, in full dev:
   
   **YAF**
   
   - `"timestampField":"start_time"`
   - `"dateFormat":"yyyy-MM-dd HH:mm:ss.S"` - need this in order for the parser to translate the non-timestamp-fromatted date/time into a proper numeric timestamp.
   - You can find the grok expressions loaded in HDFS at `/patterns/yaf`. They're also staged from the RPM installs on the local file system at `$METRON_HOME/patterns`. Anyhow, here's what YAF's Grok patterns file looks like:
   
   ```
   YAF_TIME_FORMAT %{YEAR:UNWANTED}-%{MONTHNUM:UNWANTED}-%{MONTHDAY:UNWANTED}[T ]%{HOUR:UNWANTED}:%{MINUTE:UNWANTED}:%{SECOND:UNWANTED}
   YAF_DELIMITED %{YAF_TIME_FORMAT:start_time}\|%{YAF_TIME_FORMAT:end_time}\|%{SPACE:UNWANTED}%{BASE10NUM:duration}\|%{SPACE:UNWANTED}%{BASE10NUM:rtt}\|%{SPACE:UNWANTED}%{INT:protocol}\|%{SPACE:UNWANTED}%{IP:ip_src_addr}\|%{SPACE:UNWANTED}%{INT:ip_src_port}\|%{SPACE:UNWANTED}%{IP:ip_dst_addr}\|%{SPACE:UNWANTED}%{INT:ip_dst_port}\|%{SPACE:UNWANTED}%{DATA:iflags}\|%{SPACE:UNWANTED}%{DATA:uflags}\|%{SPACE:UNWANTED}%{DATA:riflags}\|%{SPACE:UNWANTED}%{DATA:ruflags}\|%{SPACE:UNWANTED}%{WORD:isn}\|%{SPACE:UNWANTED}%{DATA:risn}\|%{SPACE:UNWANTED}%{DATA:tag}\|%{GREEDYDATA:rtag}\|%{SPACE:UNWANTED}%{INT:pkt}\|%{SPACE:UNWANTED}%{INT:oct}\|%{SPACE:UNWANTED}%{INT:rpkt}\|%{SPACE:UNWANTED}%{INT:roct}\|%{SPACE:UNWANTED}%{INT:app}\|%{GREEDYDATA:end_reason}
   ```
   
   Notice the `start_time` field - `%{YAF_TIME_FORMAT:start_time}`
   
   For an original message `"original_string": "2019-06-03 20:38:27.000|2019-06-03 20:38:27.000| 0.000| 0.000| 6| 192.168.138.158|49189| 62.75.195.236| 80| A| 0| 0| 0|9dfb1927|00000000|000|000| 1| 40| 0| 0| 0|idle ",`, this results in a Metron `timestamp` field published to the index as `"timestamp": 1559594307000`
   
   **Squid**
   
   - `"timestampField": "timestamp"`
   - No need for dateFormat bc the timestamp is in the right format for Metron OOTB.
   
   ```
   SQUID_DELIMITED %{NUMBER:timestamp}[^0-9]*%{INT:elapsed} %{IP:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url}[^0-9]*(%{IP:ip_dst_addr})?
   ```
   
   Does that answer your question?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services