[jira] [Commented] (TAJO-1222) DelimitedTextFile should be tolerant against parsing errors.

["Hyunsik Choi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TAJO-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14231824#comment-14231824 ] 

Hyunsik Choi commented on TAJO-1222:
------------------------------------

This patch enables plan-text based file formats like Json and CSV to be tolerant against parsing errors.
I added a table property {{text.error-tolerance.max-num}} to allow users to set the maximum number of
permissible parsing errors. You can set this table property as follows:

{code:sql}
CREATE EXTERNAL TABLE ... (
  ...
  ...
  model TEXT,
  screen_height INT,
  os TEXT,
  createdat TEXT,
  "data usage" TEXT
) USING JSON WITH ('text.error-tolerance.max-num' = '0') PARTITION BY
COLUMN (year TEXT, month TEXT) LOCATION 'file:///.../locket';
{code}

As I mentioned above, {{text.error-tolerance.max-num}} means the maximum number of permissible parsing errors. Additionally, it can have three different behaviors according to the number.

* If {{text.error-tolerance.max-num}} is set to negative integer (i.e., < 0), a running query will ignore all parsing errors.
* If {{text.error-tolerance.max-num}} is set to 0, a running query does not allow any parsing error.
* If {{text.error-tolerance.max-num}} is set to some positive integer number (i.e., > 0), a running query only ignores the given number of parsing errors in each task.

By default, 'text.error-tolerance.max-num' will be set to 0 if this property is not specified.

> DelimitedTextFile should be tolerant against parsing errors.
> ------------------------------------------------------------
>
>                 Key: TAJO-1222
>                 URL: https://issues.apache.org/jira/browse/TAJO-1222
>             Project: Tajo
>          Issue Type: Bug
>          Components: storage
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>             Fix For: 0.9.1
>
>
> DelimitedTextFile is a base class for plan-text file formats like CSV or JSON. In practice, due to various reasons, parsing errors are usual. But, the current implementation does not allow any parsing error. It is inconvenient in many cases.
> The objective of this issue is to enable DelimitedTextFile to tolerate parsing errors up to the number given by users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)