You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/01/13 04:29:31 UTC
[GitHub] vogievetsky opened a new issue #6850: Allow the CSV parser to configure escaping

vogievetsky opened a new issue #6850: Allow the CSV parser to configure escaping
URL: https://github.com/apache/incubator-druid/issues/6850
 
 
   I have this test Excel file here:
   https://static.imply.io/data/csv-test/EscapeTest.xlsx
   
   It is pretty simple:
   
   ![image](https://user-images.githubusercontent.com/177816/51081535-0d4cd780-16a7-11e9-9b44-1d7d4263192e.png)
   
   If you save it as CSV you get:
   
   ```
   Value,Comment,Timestamp
   3,"Lets do some ""normal"" quotes",2018-05-05T10:00:00Z
   34,"Lets do some ""normal"", quotes with comma",2018-05-06T10:00:00Z
   343,"Lets try \""it\"" with slash quotes",2018-05-07T10:00:00Z
   545,"Lets try \""it\"", with slash quotes and comma",2018-05-08T10:00:00Z
   65,Here I write \n slash n,2018-05-09T10:00:00Z
   ```
   
   Clearly when Excel saves the file it does not care for the `\` escaping. Note that newlines in Excel get saved as literal new lines... so that is a lost cause.
   
   This is an issue when trying to load it into Druid because Druid respects the `\` escapes. It would be great to have a way to specify the escaping scheme in the parser config. At least allow to opt out of the slash escapes.
   
   In this case the data gets ingested as:
   
   ![image](https://user-images.githubusercontent.com/177816/51081580-3b7ee700-16a8-11e9-9896-a61932fce711.png)
   
   See how the slashes subtly messed things up.
   
   Here is another dataset: https://static.imply.io/data/tweets/ira_small.csv
   
   When parsing it text like: `"Как говорится: \""всё течет, всё изменяется\"". Украина как всегда обвиняет Россию в собственных проблемах. #ПровокацияКиева"`
   
   Really messes up the time column:
   
   ![image](https://user-images.githubusercontent.com/177816/51081595-8ac51780-16a8-11e9-9a20-2d6b758d4f8d.png)
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org