You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by cgivre <gi...@git.apache.org> on 2016/10/11 01:37:24 UTC
[GitHub] drill pull request #607: Drill 3423:Add New HTTPD format plugin
GitHub user cgivre opened a pull request:
https://github.com/apache/drill/pull/607
Drill 3423:Add New HTTPD format plugin
This PR adds the capability to Drill to query HTTPD web server logs natively. In addition to the HTTPD parser, it also includes two UDFs: parse_url() and parse_query() which return maps of the URL and the query string respectively.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/parthchandra/drill DRILL-3423
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/607.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #607
----
commit 9d71dd29ef333404983cd400db5ecbc74614ecf6
Author: Jacques Nadeau <ja...@apache.org>
Date: 2015-07-28T22:29:32Z
DRILL-3423: Initial HTTPD log plugin. Needs tests. Would be good to improve the timestamp and cookies behaviors since we can make those more type specific.
commit 0c51192c26616ca3efbe82ffd3854ce29eac6bdf
Author: Jim Scott <ji...@13ways.com>
Date: 2015-11-03T21:43:54Z
DRILL-3423: Adding HTTPd Log Parsing functionality including full pushdown, type remapping and wildcard support.
commit c3b9dbc39f448f853515dae71c4784196382f2ca
Author: Parth Chandra <pa...@apache.org>
Date: 2016-10-11T00:55:30Z
DRILL-3423: fix merge
commit 12427e4eb4a328ed7c44c455b7142e5b6bb6c15c
Author: Jim Scott <ji...@13ways.com>
Date: 2015-11-17T20:38:40Z
DRILL-3423: Pushed through the requested columns for push down to the parser. Added more tests to cover a few more use cases. Ensured that user query fields are now completely consistent with returned values.
commit beefbfdd057cd175ed3d8b6c66dcc25c26e678e9
Author: cgivre <cg...@gmail.com>
Date: 2016-10-06T03:29:26Z
DRILL-3243: Added CSG mods. Fixed field names.
commit 43acafa5ed4d05af00480c1689e50ba96571bbde
Author: cgivre <cg...@gmail.com>
Date: 2016-10-06T03:35:37Z
DRILL-3243: Removed old test files
commit d981623c675f7dde8119dac521926b4d89ba45d0
Author: cgivre <cg...@gmail.com>
Date: 2016-10-06T03:54:52Z
DRILL-3423: Added Parse_url and parse_query() functions
commit 69fd49d6f01cc530ca1cd996aceac62abbdb574f
Author: cgivre <cg...@gmail.com>
Date: 2016-10-06T15:21:27Z
DRILL-3423: Fixed bug in parse_query()
If a query string started with an &, it would throw an error. This fix checks to see if the query string starts with an & and if so removes it.
commit d7faf1936c36141a462529de02dc4103dce8fc3f
Author: Parth Chandra <pa...@apache.org>
Date: 2016-10-07T23:24:48Z
DRILL-3423: Fix unit test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #607: Drill 3423:Add New HTTPD format plugin
Posted by chunhui-shi <gi...@git.apache.org>.
Github user chunhui-shi commented on the issue:
https://github.com/apache/drill/pull/607
Some generic questions:
Do we expect to develop one format plugin for one kind of log?
And is this format plugin designed to supporting multiple(e.g. hundreds) formats for one kind of log?
Is there any correlation functionality that allow users to define a way to correlate multiple lines and extract information from them to build a 'record'?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #607: Drill 3423:Add New HTTPD format plugin
Posted by cgivre <gi...@git.apache.org>.
Github user cgivre commented on the issue:
https://github.com/apache/drill/pull/607
@chunhui-shi
I've actually been thinking about writing a generic log parser for Drill in which the user would provide a regex with groups and a list of fields. For instance consider the following sshd log file:
```
070823 21:00:32 1 Connect root@localhost on test1
070823 21:00:48 1 Query show tables
070823 21:00:56 1 Query select * from category
070917 16:29:01 21 Query select * from location
070917 16:29:12 21 Query select * from location where id = 1 LIMIT 1
```
You can't really split this by space or tab, and dissecting it with various string slicing functions would lead to some very complex and ugly queries. But with the following regex:
```
^(\d{6}\s\d{2}:\d{2}:\d{2})\s+(\d+)\s(\w+)\s+(.+)$
```
You can extract all the fields and query them.
With respect to the HTTPD log parser, the log parser accepts a format string in the configuration (https://issues.apache.org/jira/browse/DRILL-3423) and with that you can parse any kind of HTTPD log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #607: Drill 3423:Add New HTTPD format plugin
Posted by parthchandra <gi...@git.apache.org>.
Github user parthchandra commented on the issue:
https://github.com/apache/drill/pull/607
+1. Let's get this into the 1.9 release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #607: Drill 3423:Add New HTTPD format plugin
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/607
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---