You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ditesh Kumar <ku...@gmail.com> on 2016/04/20 20:19:24 UTC

Unable to improve ListStatus performance of ParquetRelation

Hi,

When creating a DataFrame from a partitioned file structure (
sqlContext.read.parquet("s3://bucket/path/to/partitioned/parquet/filles")
), takes a lot of time to get list of files recursively from S3 when large
number of files are involved.
To circumvent this I wanted to override the FileStatusCache class
in HadoopFsRelation to create a new Relation which can fetch FileStatus
list from a cached source ( eg : MySql ). Currently this is not possible so
my questions are these :

   1. Is there any other way to do what I want to do ?
   2. If no to above, then can this extensibility be included by
   making FileStatusCache & related variable protected instead of private ?
   3. If yes to above, then can I help ?

Regards,
Ditesh Kumar

Re: Unable to improve ListStatus performance of ParquetRelation

Posted by Ted Yu <yu...@gmail.com>.
FileStatusCache used to be inside interfaces.scala

But in master branch, I no longer see it there.
Looks like refactor has removed the class.

On Wed, Apr 20, 2016 at 11:19 AM, Ditesh Kumar <ku...@gmail.com>
wrote:

> Hi,
>
> When creating a DataFrame from a partitioned file structure (
> sqlContext.read.parquet("s3://bucket/path/to/partitioned/parquet/filles")
> ), takes a lot of time to get list of files recursively from S3 when large
> number of files are involved.
> To circumvent this I wanted to override the FileStatusCache class
> in HadoopFsRelation to create a new Relation which can fetch FileStatus
> list from a cached source ( eg : MySql ). Currently this is not possible so
> my questions are these :
>
>    1. Is there any other way to do what I want to do ?
>    2. If no to above, then can this extensibility be included by
>    making FileStatusCache & related variable protected instead of private ?
>    3. If yes to above, then can I help ?
>
> Regards,
> Ditesh Kumar
>