You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ditesh Kumar <ku...@gmail.com> on 2016/04/20 20:19:24 UTC
Unable to improve ListStatus performance of ParquetRelation
Hi,
When creating a DataFrame from a partitioned file structure (
sqlContext.read.parquet("s3://bucket/path/to/partitioned/parquet/filles")
), takes a lot of time to get list of files recursively from S3 when large
number of files are involved.
To circumvent this I wanted to override the FileStatusCache class
in HadoopFsRelation to create a new Relation which can fetch FileStatus
list from a cached source ( eg : MySql ). Currently this is not possible so
my questions are these :
1. Is there any other way to do what I want to do ?
2. If no to above, then can this extensibility be included by
making FileStatusCache & related variable protected instead of private ?
3. If yes to above, then can I help ?
Regards,
Ditesh Kumar
Re: Unable to improve ListStatus performance of ParquetRelation
Posted by Ted Yu <yu...@gmail.com>.
FileStatusCache used to be inside interfaces.scala
But in master branch, I no longer see it there.
Looks like refactor has removed the class.
On Wed, Apr 20, 2016 at 11:19 AM, Ditesh Kumar <ku...@gmail.com>
wrote:
> Hi,
>
> When creating a DataFrame from a partitioned file structure (
> sqlContext.read.parquet("s3://bucket/path/to/partitioned/parquet/filles")
> ), takes a lot of time to get list of files recursively from S3 when large
> number of files are involved.
> To circumvent this I wanted to override the FileStatusCache class
> in HadoopFsRelation to create a new Relation which can fetch FileStatus
> list from a cached source ( eg : MySql ). Currently this is not possible so
> my questions are these :
>
> 1. Is there any other way to do what I want to do ?
> 2. If no to above, then can this extensibility be included by
> making FileStatusCache & related variable protected instead of private ?
> 3. If yes to above, then can I help ?
>
> Regards,
> Ditesh Kumar
>