[jira] [Work logged] (CSV-196) Store the information of raw data read by lexer

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CSV-196?focusedWorklogId=618799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618799 ]

ASF GitHub Bot logged work on CSV-196:
--------------------------------------

                Author: ASF GitHub Bot
            Created on: 05/Jul/21 20:50
            Start Date: 05/Jul/21 20:50
    Worklog Time Spent: 10m 
      Work Description: garydgregory commented on pull request #22:
URL: https://github.com/apache/commons-csv/pull/22#issuecomment-874322854


   @mattsunsjf 
   May you please rebase on master?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 618799)
    Remaining Estimate: 47h 50m  (was: 48h)
            Time Spent: 10m

> Store the information of raw data read by lexer
> -----------------------------------------------
>
>                 Key: CSV-196
>                 URL: https://issues.apache.org/jira/browse/CSV-196
>             Project: Commons CSV
>          Issue Type: Improvement
>          Components: Parser
>    Affects Versions: 1.4
>            Reporter: Matt Sun
>            Priority: Major
>              Labels: patch
>   Original Estimate: 48h
>          Time Spent: 10m
>  Remaining Estimate: 47h 50m
>
> It will be good to have CSVParser class to store the info of whether a field was enclosed by quotes in the original source file.
> For example, for this data sample:
> A, B, C
> a1, "b1", c1
> CSVParser gives us record a1, b1, c1, which is helpful because it parsed double quotes, but we also lost the information of original data at the same time. We can't tell from the CSVRecord returned whether the original data is enclosed by double quotes or not.
> In our use case, we are integrating Apache Hadoop APIs with Commons CSV.  CSV is one kind of input of Hadoop Jobs, which should support splitting input data. To accurately split a CSV file into pieces, we need to count the bytes of  data CSVParser actually read. CSVParser doesn't have accurate information of whether a field was enclosed by quotes, neither does it store raw data of the original source. Downstream users of commons CSVParser is not able to get those info.
> To suggest a fix: Extend the token/CSVRecord to have a boolean field indicating whether the column was enclosed by quotes. While Lexer is doing getNextToken, set the flag if a field is encapsulated and successfully parsed.
> I find another issue reported with similar request, but it was marked as resolved: [CSV91] https://issues.apache.org/jira/browse/CSV-91?jql=project%20%3D%20CSV%20AND%20text%20~%20%22with%20quotes%22



--
This message was sent by Atlassian Jira
(v8.3.4#803005)