You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2019/05/30 06:44:00 UTC
[jira] [Closed] (SPARK-27832) Don't decompress and create column
batch when the task is completed
[ https://issues.apache.org/jira/browse/SPARK-27832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun closed SPARK-27832.
---------------------------------
> Don't decompress and create column batch when the task is completed
> -------------------------------------------------------------------
>
> Key: SPARK-27832
> URL: https://issues.apache.org/jira/browse/SPARK-27832
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Liang-Chi Hsieh
> Priority: Major
>
> Cached relation decompresses and creates column batch when accessing cache. It's possible that a thread doesn't stop immediately reading cached relation after the task is completed. Due to race condition, cached relation might still decompresses and creates new and unnecessary batch. At the moment, the returned batch is also immediately closed. At the reader side, it can cause null exception, and we probably need to hide such exception.
> We don't need to create the batch if the task is completed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org