You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Abhishek Anand <ab...@gmail.com> on 2016/05/03 09:23:31 UTC

Re: removing header from csv file

You can use this function to remove the header from your dataset(applicable
to RDD)

def dropHeader(data: RDD[String]): RDD[String] = {
    data.mapPartitionsWithIndex((idx, lines) => {
      if (idx == 0) {
        lines.drop(1)
      }
      lines
    })
    }


Abhi

On Wed, Apr 27, 2016 at 12:55 PM, Marco Mistroni <mm...@gmail.com>
wrote:

> If u r using Scala api you can do
> Myrdd.zipwithindex.filter(_._2 >0).map(_._1)
>
> Maybe a little bit complicated but will do the trick
> As per spark CSV, you will get back a data frame which you can reconduct
> to rdd. .
> Hth
> Marco
> On 27 Apr 2016 6:59 am, "nihed mbarek" <ni...@gmail.com> wrote:
>
>> You can add a filter with string that you are sure available only in the
>> header
>>
>> Le mercredi 27 avril 2016, Divya Gehlot <di...@gmail.com> a
>> écrit :
>>
>>> yes you can remove the headers by removing the first row
>>>
>>> can first() or head() to do that
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>> On 27 April 2016 at 13:24, Ashutosh Kumar <km...@gmail.com>
>>> wrote:
>>>
>>>> I see there is a library spark-csv which can be used for removing
>>>> header and processing of csv files. But it seems it works with sqlcontext
>>>> only. Is there a way to remove header from csv files without sqlcontext ?
>>>>
>>>> Thanks
>>>> Ashutosh
>>>>
>>>
>>>
>>
>> --
>>
>> M'BAREK Med Nihed,
>> Fedora Ambassador, TUNISIA, Northern Africa
>> http://www.nihed.com
>>
>> <http://tn.linkedin.com/in/nihed>
>>
>>
>>

Re: removing header from csv file

Posted by Michael Segel <ms...@hotmail.com>.

Hi, 
Another silly question… 

Don’t you want to use the header line to help create a schema for the RDD? 

Thx

-Mike

> On May 3, 2016, at 8:09 AM, Mathieu Longtin <ma...@closetwork.org> wrote:
> 
> This only works if the files are "unsplittable". For example gzip files, each partition is one file (if you have more partitions than files), so the first line of each partition is the header.
> 
> Spark-csv extensions reads the very first line of the RDD, assumes it's the header, and then filters every occurrence of that line. Something like this (python code here, but Scala should be very similar)
> 
> header = data.first()
> data = data.filter(lambda line: line != header)
> 
> Since I had lots of small CSV files, and not all of them have the same exact header, I use the following:
> 
> file_list = sc.parallelize(list_of_csv)
> data = file_list.flatMap(function_that_reads_csvs_and_extracts_the_colums_I_want)
> 
> 
> 
> 
> On Tue, May 3, 2016 at 3:23 AM Abhishek Anand <abhis.anan007@gmail.com <ma...@gmail.com>> wrote:
> You can use this function to remove the header from your dataset(applicable to RDD)
> 
> def dropHeader(data: RDD[String]): RDD[String] = {
>     data.mapPartitionsWithIndex((idx, lines) => {
>       if (idx == 0) {
>         lines.drop(1)
>       }
>       lines
>     })
>     }
> 
> 
> Abhi 
> 
> On Wed, Apr 27, 2016 at 12:55 PM, Marco Mistroni <mmistroni@gmail.com <ma...@gmail.com>> wrote:
> If u r using Scala api you can do
> Myrdd.zipwithindex.filter(_._2 >0).map(_._1)
> 
> Maybe a little bit complicated but will do the trick
> As per spark CSV, you will get back a data frame which you can reconduct to rdd. .
> Hth
> Marco
> 
> On 27 Apr 2016 6:59 am, "nihed mbarek" <nihedmm@gmail.com <ma...@gmail.com>> wrote:
> You can add a filter with string that you are sure available only in the header 
> 
> Le mercredi 27 avril 2016, Divya Gehlot <divya.htconex@gmail.com <ma...@gmail.com>> a écrit :
> yes you can remove the headers by removing the first row 
> 
> can first() or head() to do that 
> 
> 
> Thanks,
> Divya 
> 
> On 27 April 2016 at 13:24, Ashutosh Kumar <kmr.ashutosh16@gmail.com <>> wrote:
> I see there is a library spark-csv which can be used for removing header and processing of csv files. But it seems it works with sqlcontext only. Is there a way to remove header from csv files without sqlcontext ? 
> 
> Thanks
> Ashutosh
> 
> 
> 
> -- 
> 
> M'BAREK Med Nihed,
> Fedora Ambassador, TUNISIA, Northern Africa
> http://www.nihed.com <http://www.nihed.com/>
> 
>  <http://tn.linkedin.com/in/nihed>
> 
> 
> -- 
> Mathieu Longtin
> 1-514-803-8977

Re: removing header from csv file

Posted by Mathieu Longtin <ma...@closetwork.org>.

This only works if the files are "unsplittable". For example gzip files,
each partition is one file (if you have more partitions than files), so the
first line of each partition is the header.

Spark-csv extensions reads the very first line of the RDD, assumes it's the
header, and then filters every occurrence of that line. Something like this
(python code here, but Scala should be very similar)

header = data.first()
data = data.filter(lambda line: line != header)

Since I had lots of small CSV files, and not all of them have the same
exact header, I use the following:

file_list = sc.parallelize(list_of_csv)
data =
file_list.flatMap(function_that_reads_csvs_and_extracts_the_colums_I_want)




On Tue, May 3, 2016 at 3:23 AM Abhishek Anand <ab...@gmail.com>
wrote:

> You can use this function to remove the header from your
> dataset(applicable to RDD)
>
> def dropHeader(data: RDD[String]): RDD[String] = {
>     data.mapPartitionsWithIndex((idx, lines) => {
>       if (idx == 0) {
>         lines.drop(1)
>       }
>       lines
>     })
>     }
>
>
> Abhi
>
> On Wed, Apr 27, 2016 at 12:55 PM, Marco Mistroni <mm...@gmail.com>
> wrote:
>
>> If u r using Scala api you can do
>> Myrdd.zipwithindex.filter(_._2 >0).map(_._1)
>>
>> Maybe a little bit complicated but will do the trick
>> As per spark CSV, you will get back a data frame which you can reconduct
>> to rdd. .
>> Hth
>> Marco
>> On 27 Apr 2016 6:59 am, "nihed mbarek" <ni...@gmail.com> wrote:
>>
>>> You can add a filter with string that you are sure available only in the
>>> header
>>>
>>> Le mercredi 27 avril 2016, Divya Gehlot <di...@gmail.com> a
>>> écrit :
>>>
>>>> yes you can remove the headers by removing the first row
>>>>
>>>> can first() or head() to do that
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>> On 27 April 2016 at 13:24, Ashutosh Kumar <km...@gmail.com>
>>>> wrote:
>>>>
>>>>> I see there is a library spark-csv which can be used for removing
>>>>> header and processing of csv files. But it seems it works with sqlcontext
>>>>> only. Is there a way to remove header from csv files without sqlcontext ?
>>>>>
>>>>> Thanks
>>>>> Ashutosh
>>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> M'BAREK Med Nihed,
>>> Fedora Ambassador, TUNISIA, Northern Africa
>>> http://www.nihed.com
>>>
>>> <http://tn.linkedin.com/in/nihed>
>>>
>>>
>>> --
Mathieu Longtin
1-514-803-8977