Re: SPARK-8813 - combining small files in spark sql

[Sean Owen <so...@cloudera.com>]

-user

Reynold made the comment that he thinks this was resolved by another
change; maybe he can comment.

On Thu, Jul 7, 2016 at 7:53 AM, Ajay Srivastava
<a_...@yahoo.com.invalid> wrote:
> Hi,
>
> This jira https://issues.apache.org/jira/browse/SPARK-8813 is fixed in spark
> 2.0.
> But resolution is not mentioned there.
>
> In our use case, there are big as well as many small parquet files which are
> being queried using spark sql.
> Can someone please explain what is the fix and how I can use it in spark 2.0
> ? I did search commits done in 2.0 branch and looks like I need to use
> spark.sql.files.openCostInBytes but I am not sure.
>
>
> Regards,
> Ajay

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org