[jira] [Commented] (BEAM-123) Skip header row from csv

["Daniel Halperin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225738#comment-15225738 ] 

Daniel Halperin commented on BEAM-123:
--------------------------------------

Hi Davin,

I see. We do not currently have a CSV-specific file source -- the usual guidance is to use TextIO and filter out the header manually. Obviously, I can see this is a pain.

It would not be too complicated to implement a new CSV source based on FileBasedSource, and then add logic and configuration to filter the first few lines there. The key would be making it not-splittable -- which can unfortunately hurt performance on some runners for very large files or pipelines with very few files -- so that you can know exactly which line of the input file a particular output record came from.

> Skip header row from csv 
> -------------------------
>
>                 Key: BEAM-123
>                 URL: https://issues.apache.org/jira/browse/BEAM-123
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-extensions
>            Reporter: Davin Pidoto
>            Priority: Minor
>
> Add functionality to skip header rows when reading from a csv file.
> http://stackoverflow.com/questions/28450554/skipping-header-rows-is-it-possible-with-cloud-dataflow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)