[jira] [Updated] (PHOENIX-2239) Honor strict option when catching data error during CSV load

["Gabriel Reid (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PHOENIX-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabriel Reid updated PHOENIX-2239:
----------------------------------
    Attachment: PHOENIX-2239.patch

[~jamestaylor] just to be sure/clear, you're talking about the non-MR loader (i.e. via psql) right? The MR-based loader (CsvBulkLoadTool) doesn't currently have a --strict option.

It looks like the --strict option in psql is currently only used to enforce that the list of columns coming from the CSV file matches up with columns in the table. Here's a patch that causes the ingest tool to fail if there is a failure to parse or upsert a given row as well.

Adding something like this into the MR-based loader would be more tricky, as stopping the full job from a single MR task isn't easily done I think (MR will just retry the task a few times).

> Honor strict option when catching data error during CSV load
> ------------------------------------------------------------
>
>                 Key: PHOENIX-2239
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2239
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>         Attachments: PHOENIX-2239.patch
>
>
> The Phoenix CSV Bulk Loader has a strict option (i.e. -s or --strict) which is intended to abort the load if a data error is found. Looks like we're not passing this option through the errorOnRecord method, but I think that'd be a good (and likely pretty easy) improvement:
> {code}
>         @Override
>         public void errorOnRecord(CSVRecord csvRecord, String errorMessage) {
>             LOG.error("Error upserting record {}: {}", csvRecord, errorMessage);
>         }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)