You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (JIRA)" <ji...@apache.org> on 2019/05/08 04:43:00 UTC
[jira] [Updated] (SPARK-27655) Persistent the table statistics to
metadata after fall back to hdfs
[ https://issues.apache.org/jira/browse/SPARK-27655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-27655:
--------------------------------
Attachment: disableFallBackToHdfs.png
enableFallBackToHdfs.png
> Persistent the table statistics to metadata after fall back to hdfs
> -------------------------------------------------------------------
>
> Key: SPARK-27655
> URL: https://issues.apache.org/jira/browse/SPARK-27655
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Yuming Wang
> Priority: Major
> Attachments: disableFallBackToHdfs.png, enableFallBackToHdfs.png
>
>
> It's a real case. We need to join many times from different tables, but the statistics of some tables are incorrect. So this job need 43 min, It only need 3.7min after set {{spark.sql.statistics.persistentStatsAfterFallBack=true}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org