You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Rishi Shah <ri...@gmail.com> on 2019/06/09 21:50:59 UTC

[pyspark 2.3+] Querying non-partitioned @TB data table is too slow

Hi All,

I have a table with 3TB data, stored as parquet snappy compression - 100
columns.. However I am filtering the DataFrame on date column (date between
20190501-20190530) & selecting only 20 columns & counting.. This operation
takes about 45 mins!!

Shouldn't parquet do the predicate pushdown and filtering without scanning
the entire dataset?

-- 
Regards,

Rishi Shah