You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Rishi Shah <ri...@gmail.com> on 2019/06/09 21:50:59 UTC
[pyspark 2.3+] Querying non-partitioned @TB data table is too slow
Hi All,
I have a table with 3TB data, stored as parquet snappy compression - 100
columns.. However I am filtering the DataFrame on date column (date between
20190501-20190530) & selecting only 20 columns & counting.. This operation
takes about 45 mins!!
Shouldn't parquet do the predicate pushdown and filtering without scanning
the entire dataset?
--
Regards,
Rishi Shah