You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/12/07 21:03:05 UTC

[GitHub] [drill] nielsbasjes opened a new pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

nielsbasjes opened a new pull request #2122:
URL: https://github.com/apache/drill/pull/2122


   # [DRILL-7817](https://issues.apache.org/jira/browse/DRILL-7817): Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.
   
   ## Description
   For the tool used to parse the HTTPD access log files a existing plugin exists to immediately parse the useragent string into usable parts.
   This adds this functionality as discussed with @cgivre .
   
   ## Documentation
   When enabled in the format config additional columns become available to the end user.
   If the user then queries those columns the requested values are extracted.
   
   ## Testing
   Several junit tests that show the correct behavior when enabled/disabled and when the useragent is missing from the logformat by having a test file with multiple logformats mixed.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] cgivre commented on a change in pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
cgivre commented on a change in pull request #2122:
URL: https://github.com/apache/drill/pull/2122#discussion_r540206798



##########
File path: contrib/format-httpd/README.md
##########
@@ -1,35 +1,39 @@
 # Web Server Log Format Plugin (HTTPD)
 This plugin enables Drill to read and query httpd (Apache Web Server) and nginx access logs natively. This plugin uses the work by [Niels Basjes](https://github.com/nielsbasjes
-) which is available here: https://github.com/nielsbasjes/logparser.
+) which is available here: https://github.com/nielsbasjes/logparser .

Review comment:
       Nit:  Why the extra space?

##########
File path: contrib/format-httpd/src/main/java/org/apache/drill/exec/store/httpd/HttpdParser.java
##########
@@ -35,45 +36,61 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.util.ArrayList;
 import java.util.EnumSet;
-import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.TreeMap;
+
+import static nl.basjes.parse.core.Casts.DOUBLE;
+import static nl.basjes.parse.core.Casts.DOUBLE_ONLY;
+import static nl.basjes.parse.core.Casts.LONG;
+import static nl.basjes.parse.core.Casts.LONG_ONLY;
+import static nl.basjes.parse.core.Casts.STRING;
+import static nl.basjes.parse.core.Casts.STRING_ONLY;
 
 public class HttpdParser {
 
   private static final Logger logger = LoggerFactory.getLogger(HttpdParser.class);
 
   public static final String PARSER_WILDCARD = ".*";
-  public static final String REMAPPING_FLAG = "#";
   private final Parser<HttpdLogRecord> parser;
   private final List<SchemaPath> requestedColumns;
   private final Map<String, MinorType> mappedColumns;
+  private final Map<String, Casts> columnCasts;
   private final HttpdLogRecord record;
   private final String logFormat;
+  private final boolean parseUserAgent;
+  private final String logParserRemapping;
   private Map<String, String> requestedPaths;
-  private EnumSet<Casts> casts;
-
 
-  public HttpdParser(final String logFormat, final String timestampFormat, final boolean flattenWildcards, final EasySubScan scan) {
+  public HttpdParser(
+          final String logFormat,
+          final String timestampFormat,
+          final boolean flattenWildcards,
+          final boolean parseUserAgent,
+          final String logParserRemapping,
+          final EasySubScan scan) {
 
     Preconditions.checkArgument(logFormat != null && !logFormat.trim().isEmpty(), "logFormat cannot be null or empty");
 
     this.logFormat = logFormat;
+    this.parseUserAgent = parseUserAgent;
     this.record = new HttpdLogRecord(timestampFormat, flattenWildcards);
 
-    if (timestampFormat == null) {
-      this.parser = new HttpdLoglineParser<>(HttpdLogRecord.class, logFormat);
-    } else {
-      this.parser = new HttpdLoglineParser<>(HttpdLogRecord.class, logFormat, timestampFormat);

Review comment:
       Will the parser throw errors or warnings if the `timestampFormat` is `null`?   I know you don't recommend that the user provides a custom `timestampFormat` so I was trying to optimize the code for that. 

##########
File path: contrib/format-httpd/src/main/resources/bootstrap-format-plugins.json
##########
@@ -5,7 +5,7 @@
       "formats": {
         "httpd" : {
           "type" : "httpd",
-          "logFormat" : "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"",
+          "logFormat" : "common\ncombined",

Review comment:
       Nice!  I like this a lot better than having to specify the exact config.  Can we put something in the `README` about this?  Are there other canned configs we could use?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] cgivre commented on a change in pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
cgivre commented on a change in pull request #2122:
URL: https://github.com/apache/drill/pull/2122#discussion_r540352520



##########
File path: contrib/format-httpd/src/main/java/org/apache/drill/exec/store/httpd/HttpdParser.java
##########
@@ -35,45 +36,61 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.util.ArrayList;
 import java.util.EnumSet;
-import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.TreeMap;
+
+import static nl.basjes.parse.core.Casts.DOUBLE;
+import static nl.basjes.parse.core.Casts.DOUBLE_ONLY;
+import static nl.basjes.parse.core.Casts.LONG;
+import static nl.basjes.parse.core.Casts.LONG_ONLY;
+import static nl.basjes.parse.core.Casts.STRING;
+import static nl.basjes.parse.core.Casts.STRING_ONLY;
 
 public class HttpdParser {
 
   private static final Logger logger = LoggerFactory.getLogger(HttpdParser.class);
 
   public static final String PARSER_WILDCARD = ".*";
-  public static final String REMAPPING_FLAG = "#";
   private final Parser<HttpdLogRecord> parser;
   private final List<SchemaPath> requestedColumns;
   private final Map<String, MinorType> mappedColumns;
+  private final Map<String, Casts> columnCasts;
   private final HttpdLogRecord record;
   private final String logFormat;
+  private final boolean parseUserAgent;
+  private final String logParserRemapping;
   private Map<String, String> requestedPaths;
-  private EnumSet<Casts> casts;
-
 
-  public HttpdParser(final String logFormat, final String timestampFormat, final boolean flattenWildcards, final EasySubScan scan) {
+  public HttpdParser(
+          final String logFormat,
+          final String timestampFormat,
+          final boolean flattenWildcards,
+          final boolean parseUserAgent,
+          final String logParserRemapping,
+          final EasySubScan scan) {
 
     Preconditions.checkArgument(logFormat != null && !logFormat.trim().isEmpty(), "logFormat cannot be null or empty");
 
     this.logFormat = logFormat;
+    this.parseUserAgent = parseUserAgent;
     this.record = new HttpdLogRecord(timestampFormat, flattenWildcards);
 
-    if (timestampFormat == null) {
-      this.parser = new HttpdLoglineParser<>(HttpdLogRecord.class, logFormat);
-    } else {
-      this.parser = new HttpdLoglineParser<>(HttpdLogRecord.class, logFormat, timestampFormat);

Review comment:
       Perfect!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on a change in pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on a change in pull request #2122:
URL: https://github.com/apache/drill/pull/2122#discussion_r539454963



##########
File path: contrib/format-httpd/README.md
##########
@@ -41,24 +45,67 @@ FROM dfs.test.`logfile.httpd` AS mylogs
 
 ```
 In this example, we assign an alias of `mylogs` to the table, the column name is `request_firstline_uri_query_$` and then the individual field within that mapping is `username
-`.  This particular example enables you to analyze items in query strings.  
+`.  This particular example enables you to analyze items in query strings.
 
 ### Flattening Maps
-In the event that you have a map field that you would like broken into columns rather than getting the nested fields, you can set the `flattenWildcards` option to `true` and 
-Drill will create columns for these fields.  For example if you have a URI Query option called `username`.  If you selected the `flattedWildcards` option, Drill will create a 
-field called `request_firstline_uri_query_username`.  
+In the event that you have a map field that you would like broken into columns rather than getting the nested fields, you can set the `flattenWildcards` option to `true` and
+Drill will create columns for these fields.  For example if you have a URI Query option called `username`.  If you selected the `flattedWildcards` option, Drill will create a
+field called `request_firstline_uri_query_username`.
 
-** Note that underscores in the field name are replaced with double underscores ** 
- 
- ## Useful Functions
+** Note that underscores in the field name are replaced with double underscores **
+
+## Useful Functions
  If you are using Drill to analyze web access logs, there are a few other useful functions which you should know about:
- 
+
  * `parse_url(<url>)`: This function accepts a URL as an argument and returns a map of the URL's protocol, authority, host, and path.
  * `parse_query(<query_string>)`: This function accepts a query string and returns a key/value pairing of the variables submitted in the request.
  * `parse_user_agent(<user agent>)`, `parse_user_agent( <useragent field>, <desired field> )`: The function parse_user_agent() takes a user agent string as an argument and
-  returns a map of the available fields. Note that not every field will be present in every user agent string. 
+  returns a map of the available fields. Note that not every field will be present in every user agent string.
   [Complete Docs Here](https://github.com/apache/drill/tree/master/contrib/udfs#user-agent-functions)
- 
+
+## LogParser type remapping
+**Advanced feature**
+The underlying [logparser](https://github.com/nielsbasjes/logparser) supports something called type remapping.
+Essentially it means that an extracted value which would normally be treated as an unparsable STRING can now be 'cast' to something
+that can be further cut into relevant pieces.
+
+The parameter string is a `;` separated list of mappings.
+Each mapping is a `:` separated list of
+- the name of the underlying logparser field (which is different from th Drill column name),
+- the underlying `type` which is used to determine which additional Dissectors can be applied.
+- optionally the `cast` (one of `STRING`, `LONG`, `DOUBLE`) which may impact the type of the Drill column
+
+Examples:
+- If you have a query parameter in the URL called `ua` which is really the UserAgent string and you would like to parse this you can add
+`request.firstline.uri.query.ua:HTTP.USERAGENT`
+- If you have a query parameter in the URL called `timestamp` which is really the numerical timestamp (epoch milliseconds).
+The additional "LONG" will cause the returned value be a long which tells Drill the `TIME.EPOCH` is to be interpreted as a `TIMESTAMP` column.
+`request.firstline.uri.query.timestamp:TIME.EPOCH:LONG`
+

Review comment:
       Yes, I was hoping you would like this feature to be available in Drill.
   Note that the name of the parameter does not matter. If it is called `foo` you can do `request.firstline.uri.query.foo:TIME.EPOCH:LONG` and it will work aswell.
   
   Having this is especially useful if you are parsing web analytics `beacon` logs (where the real data is all in query parameters).
   If you have a query parameter that is really the URL of your site then simply remap it to `HTTP.URI` and go from there.
   `request.firstline.uri.query.g:HTTP.URI`
   
   Another nice example of this is what I do here with the MOD_UNIQUE_ID field ... for which a built in Dissector is available:
   
   https://github.com/apache/drill/blob/29edfec469c7de4cdd621fd4914dfb143d9a2784/contrib/format-httpd/src/test/java/org/apache/drill/exec/store/httpd/TestHTTPDLogReaderUserAgent.java#L194
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2122:
URL: https://github.com/apache/drill/pull/2122#issuecomment-741637555


   Casts:
   The `Casts` are the possible types the logparser has for returning the found elements. So if it says `LONG` as one of the options you can specify a setter that receives a `Long`. This is a property that differs per field, yet the old implementation in Drill used a single value for all fields. When this value is STRING it will all work, but when I sorted the fields this changed and it all fell apart.
   
   Remapping: 
   The logparser constructs a tree of dissectors based upon types like `STRING`, `TIME.EPOCH`, `IP` and `HTTP.USERAGENT`.
   A specific Dissector is capable of parsing a certain input type into one of more pieces which will have different types than the input.
   Now in my example the `request.firstline.uri.query.ua` is a `STRING` for which no Dissector exists. 
   In the logparser you can remap this to a different type like `HTTP.USERAGENT`. If you specify this the UserAgentDissector will automatically be attached and dissect this further into the parts it can produce.
   What I see in the Drill code is that there is an option to add a '#' (REMAPPING_FLAG) and then specify a new type and this is then used to call the remapping code of the logparser. Yet I have not found any tests or documentation that show how this must be used. 
   
   I have an idea on how to make this possible.
   
   Config option:
   Yes, understood. I'll fix it.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2122:
URL: https://github.com/apache/drill/pull/2122#issuecomment-740519480


   @cgivre How should the type remapping feature be defined in drill?
   
   In the logparser you have a field like for example the query string parameter `ua` in the field `request.firstline.uri.query.ua`
   You can then remap it to a new type like `HTTP.USERAGENT` and then any Dissector that can handle the input of type `HTTP.USERAGENT` will provide additional results.
   
   So the normal flow is that you tell the logparser about the remapping 
   remap `request.firstline.uri.query.ua` to `HTTP.USERAGENT` 
   and then if you ask for all possible columns it should give you a lot more than before.
   So in the end I expect to have Drill columns like (with the flatten wildcards enabled)
   `request_firstline_uri_query_ua_agent__name__version__major`
   
   How do I do that? I have not been able to figure it out from the code and documentation.
    


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] cgivre commented on a change in pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
cgivre commented on a change in pull request #2122:
URL: https://github.com/apache/drill/pull/2122#discussion_r539412478



##########
File path: contrib/format-httpd/README.md
##########
@@ -41,24 +45,67 @@ FROM dfs.test.`logfile.httpd` AS mylogs
 
 ```
 In this example, we assign an alias of `mylogs` to the table, the column name is `request_firstline_uri_query_$` and then the individual field within that mapping is `username
-`.  This particular example enables you to analyze items in query strings.  
+`.  This particular example enables you to analyze items in query strings.
 
 ### Flattening Maps
-In the event that you have a map field that you would like broken into columns rather than getting the nested fields, you can set the `flattenWildcards` option to `true` and 
-Drill will create columns for these fields.  For example if you have a URI Query option called `username`.  If you selected the `flattedWildcards` option, Drill will create a 
-field called `request_firstline_uri_query_username`.  
+In the event that you have a map field that you would like broken into columns rather than getting the nested fields, you can set the `flattenWildcards` option to `true` and
+Drill will create columns for these fields.  For example if you have a URI Query option called `username`.  If you selected the `flattedWildcards` option, Drill will create a
+field called `request_firstline_uri_query_username`.
 
-** Note that underscores in the field name are replaced with double underscores ** 
- 
- ## Useful Functions
+** Note that underscores in the field name are replaced with double underscores **
+
+## Useful Functions
  If you are using Drill to analyze web access logs, there are a few other useful functions which you should know about:
- 
+
  * `parse_url(<url>)`: This function accepts a URL as an argument and returns a map of the URL's protocol, authority, host, and path.
  * `parse_query(<query_string>)`: This function accepts a query string and returns a key/value pairing of the variables submitted in the request.
  * `parse_user_agent(<user agent>)`, `parse_user_agent( <useragent field>, <desired field> )`: The function parse_user_agent() takes a user agent string as an argument and
-  returns a map of the available fields. Note that not every field will be present in every user agent string. 
+  returns a map of the available fields. Note that not every field will be present in every user agent string.
   [Complete Docs Here](https://github.com/apache/drill/tree/master/contrib/udfs#user-agent-functions)
- 
+
+## LogParser type remapping
+**Advanced feature**
+The underlying [logparser](https://github.com/nielsbasjes/logparser) supports something called type remapping.
+Essentially it means that an extracted value which would normally be treated as an unparsable STRING can now be 'cast' to something
+that can be further cut into relevant pieces.
+
+The parameter string is a `;` separated list of mappings.
+Each mapping is a `:` separated list of
+- the name of the underlying logparser field (which is different from th Drill column name),
+- the underlying `type` which is used to determine which additional Dissectors can be applied.
+- optionally the `cast` (one of `STRING`, `LONG`, `DOUBLE`) which may impact the type of the Drill column
+
+Examples:
+- If you have a query parameter in the URL called `ua` which is really the UserAgent string and you would like to parse this you can add
+`request.firstline.uri.query.ua:HTTP.USERAGENT`
+- If you have a query parameter in the URL called `timestamp` which is really the numerical timestamp (epoch milliseconds).
+The additional "LONG" will cause the returned value be a long which tells Drill the `TIME.EPOCH` is to be interpreted as a `TIMESTAMP` column.
+`request.firstline.uri.query.timestamp:TIME.EPOCH:LONG`
+

Review comment:
       This is slick! 🥇 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] cgivre commented on pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
cgivre commented on pull request #2122:
URL: https://github.com/apache/drill/pull/2122#issuecomment-740768976


   > So I tried to fix the test which was really hard because the ordering of the columns in case of a wildcard was based on the hash (i.e. visually random).
   > 
   > * I made the ordering of the columns in case of the wildcard a stable textual ordering (by means of a TreeMap instead of a HashMap).
   > * I found that the Casts were the same for all columns (which is incorrect, so I fixed that).
   
   Can you clarify this?  The unit tests do verify that Drill is producing the correct data types and they seemed to be working. 
   
   > * As a consequence the tests become much easier to read because everything is sorted.
   
   :-)
   
   > 
   > At this point I think the type remapping doesn't work.
   > I haven't seen any testing code or documentation so I'm not sure.
   
   Can you clarify which type remapping you're referring to?  When we did the refactoring, I added unit tests that check for data types.  Most fields are strings, so they get mapped to `VARCHAR` types.  However, the timestamps and dates get mapped to the appropriate data types.  Also, the pieces of the times also get mapped to `DOUBLE` data types. (I think).
   
   You can confirm that in each unit test where it constructs a schema.
   
   https://github.com/apache/drill/blob/7a614d8ce7a0018c6ffaa12955d56366ee62a6d6/contrib/format-httpd/src/test/java/org/apache/drill/exec/store/httpd/TestHTTPDLogReader.java#L224-L228
   
   > Because of the lack of testing I'm pretty sure one of my changes broke it.
   > @cgivre If you can please give me a hint on how you intended it to work I can pick that up for a test to include.
   
   I actually liked your original implementation with the config option.  What I was thinking however was that if the option was enabled, dissector would only be added if the user executed a star query OR they included one of the raw `user-agent` fields in the query. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2122:
URL: https://github.com/apache/drill/pull/2122#issuecomment-740733560


   So I tried to fix the test which was really hard because the ordering of the columns in case of a wildcard was based on the hash (i.e. visually random).
   
   - I made the ordering of the columns in case of the wildcard a stable textual ordering (by means of a TreeMap instead of a HashMap).
   - I found that the Casts were the same for all columns (which is incorrect, so I fixed that).
   - As a consequence the tests become much easier to read because everything is sorted.
   
   At this point I think the type remapping doesn't work.
   I haven't seen any testing code or documentation so I'm not sure.
   Because of the lack of testing I'm pretty sure one of my changes broke it.
   @cgivre If you can please give me a hint on how you intended it to work I can pick that up for a test to include.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes edited a comment on pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes edited a comment on pull request #2122:
URL: https://github.com/apache/drill/pull/2122#issuecomment-741637555


   Casts:
   The `Casts` are the possible types the logparser has for returning the found elements. So if it says `LONG` as one of the options you can specify a setter that receives a `Long`. This is a property that differs per field, yet the old implementation in Drill used a single value for all fields. When this value is STRING it will all work, but when I sorted the fields this changed and it all fell apart.
   
   Remapping: 
   The logparser constructs a tree of dissectors based upon types like `STRING`, `TIME.EPOCH`, `IP` and `HTTP.USERAGENT`.
   A specific Dissector is capable of parsing a certain input type into one of more pieces which will have different types than the input.
   Now in my example the `request.firstline.uri.query.ua` is a `STRING` for which no Dissector exists. 
   In the logparser you can remap this to a different type like `HTTP.USERAGENT`. If you specify this the UserAgentDissector will automatically be attached and dissect this further into the parts it can produce.
   What I see in the Drill code is that there is an option to add a '#' (REMAPPING_FLAG) and then specify a new type and this is then used to call the remapping code of the logparser. Yet I have not found any tests or documentation that show how this must be used. 
   
   I have an idea on how to make this possible.
   
   Config option:
   If you enable the useragent parser and then you do not ask for any fields from it the following happens.
   1) An instance of UserAgentDissector is created to see what can be parsed (takes about 1 second).
   2) If then no fields are asked from it this instance of the UserAgentDissector is simply removed from the parsing again. 
   So during the actually parsing run there is no performance impact, only during the startup phase.
   
   I'll improve this.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on a change in pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on a change in pull request #2122:
URL: https://github.com/apache/drill/pull/2122#discussion_r537874370



##########
File path: contrib/format-httpd/README.md
##########
@@ -10,15 +10,16 @@ There are five fields which you can to configure in order for Drill to read web
 * **`extensions`**:  The file extension of your web server logs.  Defaults to `httpd`.
 * **`maxErrors`**:  Sets the plugin error tolerance. When set to any value less than `0`, Drill will ignore all errors. If unspecified then maxErrors is 0 which will cause the query to fail on the first error.
 * **`flattenWildcards`**: There are a few variables which Drill extracts into maps.  Defaults to `false`.
-
+* **`enableUserAgentParser`**:  This enables the additional Yauaa useragent parser. If unspecified this is disabled because there is a noticeable startup overhead of this plugin, even if not used.
 

Review comment:
       It might work.
   
   The underlying logparser works on the idea of a type. The useragent parser hooks to the "HTTP.USERAGENT" type and allows dissecting anything of that type, regardless of the name. 
   
   Practically speaking:
   The logparser code only maps the request header "user-agent" to field "request.user-agent" with type "HTTP.USERAGENT"
   So unless someone uses the type remapping to extract the useragent from a different field this is the only naturally occurring place.
   
   If you do it that way you are working under an assumption about the name of the column instead of the internal types.
   
   You then also have to take into account that these do not need the extra step:
   
       request_user-agent
       request_user-agent_last
   
   and these do
   
       request_user-agent_operating__system__version__major
       request_user-agent_last_operating__system__version__major
   
   So what I think is that you can activate it automatically if someone does `*` or asks for something as mentioned above.
   Yet I do think that if any type remapping is done it should also be activated if something is mapped to the input type of the UserAgentDissector.
   
   If there are cases where you activate the plugin when it is not needed then the only downside of this is that you'll have extra startup time and extra memory usage.
   
   The actual parsing speed is not affected because the logparser will simply not do any of the Yauaa code if it not needed.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes edited a comment on pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes edited a comment on pull request #2122:
URL: https://github.com/apache/drill/pull/2122#issuecomment-740519480


   @cgivre How should the type remapping feature be defined in drill?
   
   In the logparser you have a field like for example the query string parameter `ua` in the field `request.firstline.uri.query.ua`
   You can then remap it to a new type like `HTTP.USERAGENT` and then any Dissector that can handle the input of type `HTTP.USERAGENT` will provide additional results.
   
   So the normal flow is that you tell the logparser about the remapping 
   remap `request.firstline.uri.query.ua` to `HTTP.USERAGENT` 
   and then if you ask for all possible columns it should give you a lot more than before.
   So in the end I expect to have Drill columns like (with the flatten wildcards enabled)
   `request_firstline_uri_query_ua_agent__name__version__major`
   
   How do I do that? I have not been able to figure it out from the code and documentation.
   
   I added test at my end to illustrate:
   https://github.com/nielsbasjes/yauaa/blob/master/udfs/logparser/src/test/java/nl/basjes/parse/useragent/dissector/TestLoadDissectorDynamicallyInPig.java#L52
   Parsing this logfile https://github.com/nielsbasjes/yauaa/blob/master/udfs/logparser/src/test/resources/access.log
   Here the `ua` query parameter is parsed as a useragent.
   
   
    


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on a change in pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on a change in pull request #2122:
URL: https://github.com/apache/drill/pull/2122#discussion_r539471783



##########
File path: contrib/format-httpd/README.md
##########
@@ -41,24 +45,67 @@ FROM dfs.test.`logfile.httpd` AS mylogs
 
 ```
 In this example, we assign an alias of `mylogs` to the table, the column name is `request_firstline_uri_query_$` and then the individual field within that mapping is `username
-`.  This particular example enables you to analyze items in query strings.  
+`.  This particular example enables you to analyze items in query strings.
 
 ### Flattening Maps
-In the event that you have a map field that you would like broken into columns rather than getting the nested fields, you can set the `flattenWildcards` option to `true` and 
-Drill will create columns for these fields.  For example if you have a URI Query option called `username`.  If you selected the `flattedWildcards` option, Drill will create a 
-field called `request_firstline_uri_query_username`.  
+In the event that you have a map field that you would like broken into columns rather than getting the nested fields, you can set the `flattenWildcards` option to `true` and
+Drill will create columns for these fields.  For example if you have a URI Query option called `username`.  If you selected the `flattedWildcards` option, Drill will create a
+field called `request_firstline_uri_query_username`.
 
-** Note that underscores in the field name are replaced with double underscores ** 
- 
- ## Useful Functions
+** Note that underscores in the field name are replaced with double underscores **
+
+## Useful Functions
  If you are using Drill to analyze web access logs, there are a few other useful functions which you should know about:
- 
+
  * `parse_url(<url>)`: This function accepts a URL as an argument and returns a map of the URL's protocol, authority, host, and path.
  * `parse_query(<query_string>)`: This function accepts a query string and returns a key/value pairing of the variables submitted in the request.
  * `parse_user_agent(<user agent>)`, `parse_user_agent( <useragent field>, <desired field> )`: The function parse_user_agent() takes a user agent string as an argument and
-  returns a map of the available fields. Note that not every field will be present in every user agent string. 
+  returns a map of the available fields. Note that not every field will be present in every user agent string.
   [Complete Docs Here](https://github.com/apache/drill/tree/master/contrib/udfs#user-agent-functions)
- 
+
+## LogParser type remapping
+**Advanced feature**
+The underlying [logparser](https://github.com/nielsbasjes/logparser) supports something called type remapping.
+Essentially it means that an extracted value which would normally be treated as an unparsable STRING can now be 'cast' to something
+that can be further cut into relevant pieces.
+
+The parameter string is a `;` separated list of mappings.
+Each mapping is a `:` separated list of
+- the name of the underlying logparser field (which is different from th Drill column name),
+- the underlying `type` which is used to determine which additional Dissectors can be applied.
+- optionally the `cast` (one of `STRING`, `LONG`, `DOUBLE`) which may impact the type of the Drill column
+
+Examples:
+- If you have a query parameter in the URL called `ua` which is really the UserAgent string and you would like to parse this you can add
+`request.firstline.uri.query.ua:HTTP.USERAGENT`
+- If you have a query parameter in the URL called `timestamp` which is really the numerical timestamp (epoch milliseconds).
+The additional "LONG" will cause the returned value be a long which tells Drill the `TIME.EPOCH` is to be interpreted as a `TIMESTAMP` column.
+`request.firstline.uri.query.timestamp:TIME.EPOCH:LONG`
+

Review comment:
       Have a look at this file as an example of what I mean with a `beacon` log:
   https://github.com/nielsbasjes/logparser/blob/master/httpdlog/httpdlog-pigloader/src/test/resources/omniture-access.log
   Here the `g` query parameter is the real URL ... which in itself has query parameters of its own that I'm interested in for my analysis.
   
   This is the kind of logfile that actually triggered me to write this library all those years ago.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2122:
URL: https://github.com/apache/drill/pull/2122#issuecomment-740651052


   While fixing a test that now includes many more columns I found the ordering of the columns to be "randomized" because they are put into a hashmap... which means sorting by the hash of the key string ...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on pull request #2122:
URL: https://github.com/apache/drill/pull/2122#issuecomment-741837747


   Note: I found an edgecase bug in logparser that occurs only in very rare cases. 
   Already fixed it and released version 5.7


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] nielsbasjes commented on a change in pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
nielsbasjes commented on a change in pull request #2122:
URL: https://github.com/apache/drill/pull/2122#discussion_r540301492



##########
File path: contrib/format-httpd/README.md
##########
@@ -1,35 +1,39 @@
 # Web Server Log Format Plugin (HTTPD)
 This plugin enables Drill to read and query httpd (Apache Web Server) and nginx access logs natively. This plugin uses the work by [Niels Basjes](https://github.com/nielsbasjes
-) which is available here: https://github.com/nielsbasjes/logparser.
+) which is available here: https://github.com/nielsbasjes/logparser .
 
 ## Configuration
-There are five fields which you can to configure in order for Drill to read web server logs.  In general the defaults should be fine, however the fields are:
+There are several fields which you can specify in order for Drill to read web server logs. In general the defaults should be fine, however the fields are:
 * **`logFormat`**:  The log format string is the format string found in your web server configuration. If you have multiple logFormats then you can add all of them in this
  single parameter separated by a newline (`\n`). The parser will automatically select the first matching format.
+ Note that the well known formats `common`, `combined`, `combinedio`, `referer` and `agent` are also accepted as logFormat.

Review comment:
       Here I already put some documentation about the builtin formats. Is this enough?

##########
File path: contrib/format-httpd/README.md
##########
@@ -1,35 +1,39 @@
 # Web Server Log Format Plugin (HTTPD)
 This plugin enables Drill to read and query httpd (Apache Web Server) and nginx access logs natively. This plugin uses the work by [Niels Basjes](https://github.com/nielsbasjes
-) which is available here: https://github.com/nielsbasjes/logparser.
+) which is available here: https://github.com/nielsbasjes/logparser .

Review comment:
       Yes, sorry. I thought it would break the link.

##########
File path: contrib/format-httpd/src/main/java/org/apache/drill/exec/store/httpd/HttpdParser.java
##########
@@ -35,45 +36,61 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.util.ArrayList;
 import java.util.EnumSet;
-import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.TreeMap;
+
+import static nl.basjes.parse.core.Casts.DOUBLE;
+import static nl.basjes.parse.core.Casts.DOUBLE_ONLY;
+import static nl.basjes.parse.core.Casts.LONG;
+import static nl.basjes.parse.core.Casts.LONG_ONLY;
+import static nl.basjes.parse.core.Casts.STRING;
+import static nl.basjes.parse.core.Casts.STRING_ONLY;
 
 public class HttpdParser {
 
   private static final Logger logger = LoggerFactory.getLogger(HttpdParser.class);
 
   public static final String PARSER_WILDCARD = ".*";
-  public static final String REMAPPING_FLAG = "#";
   private final Parser<HttpdLogRecord> parser;
   private final List<SchemaPath> requestedColumns;
   private final Map<String, MinorType> mappedColumns;
+  private final Map<String, Casts> columnCasts;
   private final HttpdLogRecord record;
   private final String logFormat;
+  private final boolean parseUserAgent;
+  private final String logParserRemapping;
   private Map<String, String> requestedPaths;
-  private EnumSet<Casts> casts;
-
 
-  public HttpdParser(final String logFormat, final String timestampFormat, final boolean flattenWildcards, final EasySubScan scan) {
+  public HttpdParser(
+          final String logFormat,
+          final String timestampFormat,
+          final boolean flattenWildcards,
+          final boolean parseUserAgent,
+          final String logParserRemapping,
+          final EasySubScan scan) {
 
     Preconditions.checkArgument(logFormat != null && !logFormat.trim().isEmpty(), "logFormat cannot be null or empty");
 
     this.logFormat = logFormat;
+    this.parseUserAgent = parseUserAgent;
     this.record = new HttpdLogRecord(timestampFormat, flattenWildcards);
 
-    if (timestampFormat == null) {
-      this.parser = new HttpdLoglineParser<>(HttpdLogRecord.class, logFormat);
-    } else {
-      this.parser = new HttpdLoglineParser<>(HttpdLogRecord.class, logFormat, timestampFormat);

Review comment:
       No, the parser already handles this case: https://github.com/nielsbasjes/logparser/blob/master/httpdlog/httpdlog-parser/src/main/java/nl/basjes/parse/httpdlog/dissectors/TimeStampDissector.java#L65

##########
File path: contrib/format-httpd/src/main/resources/bootstrap-format-plugins.json
##########
@@ -5,7 +5,7 @@
       "formats": {
         "httpd" : {
           "type" : "httpd",
-          "logFormat" : "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"",
+          "logFormat" : "common\ncombined",

Review comment:
       I already did that. Please check if it is enough documentation (and clear enough for others).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] cgivre merged pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
cgivre merged pull request #2122:
URL: https://github.com/apache/drill/pull/2122


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [drill] cgivre commented on a change in pull request #2122: [DRILL-7817] Allow immediate parsing of the useragent from within the HTTPD LogFormat format plugin.

Posted by GitBox <gi...@apache.org>.
cgivre commented on a change in pull request #2122:
URL: https://github.com/apache/drill/pull/2122#discussion_r537838475



##########
File path: contrib/format-httpd/README.md
##########
@@ -10,15 +10,16 @@ There are five fields which you can to configure in order for Drill to read web
 * **`extensions`**:  The file extension of your web server logs.  Defaults to `httpd`.
 * **`maxErrors`**:  Sets the plugin error tolerance. When set to any value less than `0`, Drill will ignore all errors. If unspecified then maxErrors is 0 which will cause the query to fail on the first error.
 * **`flattenWildcards`**: There are a few variables which Drill extracts into maps.  Defaults to `false`.
-
+* **`enableUserAgentParser`**:  This enables the additional Yauaa useragent parser. If unspecified this is disabled because there is a noticeable startup overhead of this plugin, even if not used.
 

Review comment:
       I wonder if we could do something here to only activate this function if:
   1.  The user executes a star query OR
   2.  They specified a field with `user-agent` in the field name?
   
   Do you think that would work?  
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org