[jira] [Created] (DRILL-8457) Allow configuring csv parser in http storage plugin configuration

["Zbigniew Tomanek (Jira)" <ji...@apache.org>]

Zbigniew Tomanek created DRILL-8457:
---------------------------------------

             Summary: Allow configuring csv parser in http storage plugin configuration
                 Key: DRILL-8457
                 URL: https://issues.apache.org/jira/browse/DRILL-8457
             Project: Apache Drill
          Issue Type: Improvement
          Components: Storage - HTTP
    Affects Versions: Future
            Reporter: Zbigniew Tomanek
             Fix For: Future


Currently there is no way to configure csv parser when http plugin is used. Because of that some kind of files cannot be parsed (e.g. when any column has more than 4096 chars or file has a delimiter different from `,`).

Since in DataWalk we utilize http plugin quite often we've changed our internal fork of Drill so following parser/format properties can be configured using additional `csvOptions` field:

 ```json
{
 "csvOptions": {
          "delimiter": "\t",
          "quote": "\"",
          "quote_escape": "\"",
          "line_separator": "\n",
          "header_extraction_enabled": null,
          "number_of_rows_to_skip": 0,
          "number_of_records_to_read": -1,
          "line_separator_detection_enabled": true,
          "max_columns": 512,
          "max_chars_per_column": 4096,
          "skip_empty_lines": true,
          "ignore_leading_whitespaces": true,
          "ignore_trailing_whitespaces": true,
          "null_value": null
        }
}
```

I'd be glad to get feedback whether creating PR with these changes would bring any value to the Drill



--
This message was sent by Atlassian Jira
(v8.20.10#820010)