You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by amin mohebbi <am...@yahoo.com.INVALID> on 2018/07/20 01:34:40 UTC

Parquet

We do have two big tables each includes 5 billion of rows, so my question here is should we partition /sort the data and convert it to Parquet before doing any join?
Best Regards ....................................................... Amin Mohebbi PhD candidate in Software Engineering   at university of Malaysia   Tel : +60 18 2040 017 E-Mail : TP025921@ex.apiit.edu.my               amin_524@me.com

Re: Parquet

Posted by Muthu Jayakumar <ba...@gmail.com>.

I generally write to Parquet when I want to repeat the operation of reading
data and perform different operations on it every time. This would save db
time for me.

Thanks
Muthu

On Thu, Jul 19, 2018, 18:34 amin mohebbi <am...@yahoo.com.invalid>
wrote:

> We do have two big tables each includes 5 billion of rows, so my question
> here is should we partition /sort the data and convert it to Parquet before
> doing any join?
>
> Best Regards ....................................................... Amin
> Mohebbi PhD candidate in Software Engineering   at university of Malaysia
> Tel : +60 18 2040 017 E-Mail : TP025921@ex.apiit.edu.my
> amin_524@me.com
>