You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Shuporno Choudhury <sh...@gmail.com> on 2018/04/23 06:03:09 UTC

Getting Corrupt Records while loading data into dataframe from csv file

Hi all,

I have a manually created schema using which I am loading data from
multiple csv files to a dataframe.
Now, if there are certain records that fail the provided schema, is there a
way to get those rejected records and continue with the process of loading
data into the dataframe?
As of now, it seems the options that I have the are the 3 modes
(PERMISSIVE, DROPMALFORMED and FAILFAST), none of which seem to fulfill the
objective.


-- 
--Thanks,
Shuporno Choudhury