[jira] [Updated] (GOBBLIN-231) Grok to Json Converter

["Tamas Nemeth (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tamas Nemeth updated GOBBLIN-231:
---------------------------------
    Priority: Major  (was: Minor)

> Grok to Json Converter
> ----------------------
>
>                 Key: GOBBLIN-231
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-231
>             Project: Apache Gobblin
>          Issue Type: New Feature
>          Components: gobblin-core
>            Reporter: Tamas Nemeth
>            Assignee: Abhishek Tiwari
>
> Converter can convert text to json base on a GROK pattern.
> GrokToJsonConverter accepts already deserialized text row, String.
> Converts Text to JSON based on Grok pattern. Schema is represented by the form of JsonArray same interface being used by CsvToJonConverter.
> Each text record is represented by a String.
> The converter only supports Grok patterns where every group is named because it uses the group names as column names.
> The following config properties can be set:
> The grok pattern to use for the conversion:
> converter.grok_to_json.pattern=^%{IPORHOST:clientip} (?:-|%{USER:ident}) (?:-|%{USER:auth}) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\" %{NUMBER:response} (?:-|%{NUMBER:bytes})
> Path to the grok patterns (if not set it will use the default ones):
> converter.grok_to_json.patterns=/tmp/grok_patterns
> Treat empty string as null value:
> converter.grok_to_json.empty_as_null=true
> Specify the null string:
> converter.grok_to_json.null_string=null
> Example of schema:
>  [
>   {
>     "columnName": "Day",
>     "comment": "",
>     "isNullable": "true",
>     "dataType": {
>       "type": "string"
>     }
>   },
>   {
>     "columnName": "Pageviews",
>     "comment": "",
>     "isNullable": "true",
>     "dataType": {
>       "type": "long"
>     }
>   }
> ]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)