[jira] [Commented] (SPARK-33488) Re SPARK-21820. Creating Spark dataframe with carriage return/line feed leaves cr in multiline

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-33488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237855#comment-17237855 ] 

Hyukjin Kwon commented on SPARK-33488:
--------------------------------------

In Spark now you can set lineSep option

> Re SPARK-21820.  Creating Spark dataframe with carriage return/line feed leaves cr in multiline
> -----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-33488
>                 URL: https://issues.apache.org/jira/browse/SPARK-33488
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.4.5
>         Environment: Apache 2.4.5
> Databricks 6.6
> Spark-NLP 2.6.3
>            Reporter: Greg Werner
>            Priority: Major
>
> In SPARK-21820 I see what seems to be the same issue reported, but marked as resolved there.  Over the past few days I have battled a dataset that occasionally has \r\n at the end of lines and I claim I do see this errant behavior of not removing \r\n.
> In my code, I do 
> {code:java}
> // code placeholder# CSV options
> infer_schema = "false"
> first_row_is_header = "true"
> multi_line = "true"
> delimiter = ","
> # The applied options are for CSV files. For other file types, these will be ignored.
> df_train = spark.read.format(train_file_type) \
>   .option("inferSchema", infer_schema) \
>   .option("header", first_row_is_header) \
>   .option("sep", delimiter) \
>   .option("multiLine", multi_line) \
>   .option("escape", '"') \
>   .load(train_file_location)
> {code}
> So I am reading in a csv file and setting multiLine to true.  However, all cases where there are \r\n in the training_file, \r is left behind.  This includes the header which has a column ending in \r.  The only way I have been able to workaround this is to manually edit the data file to remove the \r, but I do not want to do this on a case to case basis.
> Therefore, I am claiming this behavior is still present in 2.4.5 and is a bug.
> I am using version 2.4.5 because I am using Spark-NLP which to my knowledge has not been built to use 3 yet, so the version is key for me.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org