You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/07/17 18:02:00 UTC

[jira] [Updated] (SPARK-48928) Log Warning for Calling .unpersist() on Locally Checkpointed RDDs

     [ https://issues.apache.org/jira/browse/SPARK-48928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-48928:
-----------------------------------
    Labels: pull-request-available  (was: )

> Log Warning for Calling .unpersist() on Locally Checkpointed RDDs
> -----------------------------------------------------------------
>
>                 Key: SPARK-48928
>                 URL: https://issues.apache.org/jira/browse/SPARK-48928
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.0.0
>            Reporter: Mingkang Li
>            Priority: Minor
>              Labels: pull-request-available
>
> *Summary:* 
> This change proposes to log a warning message when the {{.unpersist()}} method is called on RDDs that have been locally checkpointed in Apache Spark. This aims to inform users about the potential risks of unpersisting such RDDs without altering the existing behavior of the method.
> *Background:*
>  Local checkpointing in Spark truncates the lineage of an RDD, meaning that the RDD cannot be recomputed from its source. If an RDD that has been locally checkpointed is unpersisted, it loses its data and cannot be regenerated. This can lead to job failures if subsequent actions or transformations are attempted on the unpersisted RDD.
> *Proposed Change:* 
> To mitigate this issue, a warning message will be logged whenever {{.unpersist()}} is called on a locally checkpointed RDD. This approach maintains the current functionality while alerting users to the potential consequences of their actions. This change is intended to be non-disruptive and is a step towards better user awareness and debugging.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org