You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/09/08 13:12:00 UTC
[jira] [Commented] (FLINK-7050) RFC Compliant CSV Parser for Table
Source
[ https://issues.apache.org/jira/browse/FLINK-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158613#comment-16158613 ]
ASF GitHub Bot commented on FLINK-7050:
---------------------------------------
GitHub user uybhatti opened a pull request:
https://github.com/apache/flink/pull/4660
[FLINK-7050][table] Add support of RFC compliant CSV parser for Table Source
## What is the purpose of the change
Currently CsvInputFormat is not compliant with RFC 4180 standards and we can't correctly parse fields containing double quotes, line delimiter or field delimiter.
## Brief change log
- RFCCsvInputFormat is added to support RFC 4180 standards.
- Also we added RFCRowCsvInputFormat, RFCTupleCsvInputFormat and RFCCsvTableSource for this purpose.
- New dependency for Jackson parser is added in pom.xml of flink-table.
## Verifying this change
This change added tests and can be verified as follows:
- RFCCsvInputFormatTest and RFCRowCsvInputFormatTest are added to test the added CSV Input Format functionality.
- TableSourceTest#testRFCCsvTableSourceBuilder is added for RFCCsvTableSource.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **yes**
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **yes**
## Documentation
- Does this pull request introduce a new feature? **yes**
- If yes, how is the feature documented? **not documented yet**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uybhatti/flink FLINK-7050
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4660.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4660
----
commit 977da4649ab01b3c924640c03883312a5c2f1427
Author: uybhatti <uy...@gmail.com>
Date: 2017-09-07T14:53:24Z
[FLINK-7050][table] Add support of RFC compliant CSV parser for Table Source
----
> RFC Compliant CSV Parser for Table Source
> -----------------------------------------
>
> Key: FLINK-7050
> URL: https://issues.apache.org/jira/browse/FLINK-7050
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
> Affects Versions: 1.3.1
> Reporter: Usman Younas
> Assignee: Usman Younas
> Labels: csv, parsing
> Fix For: 1.4.0
>
>
> Currently, Flink CSV parser is not compliant with RFC 4180. Due to this issue, it was not able to parse standard csv files including double quotes and delimiters with in fields etc.
> In order to produce this bug, we can take a csv file with double quotes included in field of the records and parse it using Flink CSV parser. One of the issue is mentioned in the jira [FLINK-4785|https://issues.apache.org/jira/browse/FLINK-4785].
> The CSV related issues will be solved by making CSV parser compliant with RFC 4180 standards for Table Source.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)