You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by matd <ma...@gmail.com> on 2015/08/17 10:59:39 UTC
S3n, parallelism, partitions
Hello,
I would like to understand how the work is parallelized accross a Spark
cluster (and what is left to the driver) when I read several files from a
single folder in s3 "s3n://bucket_xyz/some_folder_having_many_files_in_it/"
How files (or file parts) are mapped to partitions ?
Thanks
Mathieu
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/S3n-parallelism-partitions-tp24293.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: S3n, parallelism, partitions
Posted by Akshat Aranya <aa...@gmail.com>.
This will also depend on the file format you are using.
A word of advice: you would be much better off with the s3a file system.
As I found out recently the hard way, s3n has some issues with reading
through entire files even when looking for headers.
On Mon, Aug 17, 2015 at 2:10 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> s3n underneath uses the hadoop api, so i guess it would partition
> according to your hadoop configuration (128MB per partition by default)
>
> Thanks
> Best Regards
>
> On Mon, Aug 17, 2015 at 2:29 PM, matd <ma...@gmail.com> wrote:
>
>> Hello,
>>
>> I would like to understand how the work is parallelized accross a Spark
>> cluster (and what is left to the driver) when I read several files from a
>> single folder in s3
>> "s3n://bucket_xyz/some_folder_having_many_files_in_it/"
>>
>> How files (or file parts) are mapped to partitions ?
>>
>> Thanks
>> Mathieu
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/S3n-parallelism-partitions-tp24293.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: S3n, parallelism, partitions
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
s3n underneath uses the hadoop api, so i guess it would partition according
to your hadoop configuration (128MB per partition by default)
Thanks
Best Regards
On Mon, Aug 17, 2015 at 2:29 PM, matd <ma...@gmail.com> wrote:
> Hello,
>
> I would like to understand how the work is parallelized accross a Spark
> cluster (and what is left to the driver) when I read several files from a
> single folder in s3 "s3n://bucket_xyz/some_folder_having_many_files_in_it/"
>
> How files (or file parts) are mapped to partitions ?
>
> Thanks
> Mathieu
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/S3n-parallelism-partitions-tp24293.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>