You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2024/11/03 22:32:00 UTC
[jira] [Resolved] (SPARK-50151) RocksDB Hardening: Fix new file mapping version advancement and ineffective file reuse bug
[ https://issues.apache.org/jira/browse/SPARK-50151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-50151.
----------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 48685
[https://github.com/apache/spark/pull/48685]
> RocksDB Hardening: Fix new file mapping version advancement and ineffective file reuse bug
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-50151
> URL: https://issues.apache.org/jira/browse/SPARK-50151
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: B. Micheal Okutubo
> Assignee: B. Micheal Okutubo
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> There are 2 bugs in the recently added new approach for RocksDB SST file mapping in this PR: [https://github.com/apache/spark/pull/47875]
> # The file mapping version is not properly advancing and only advances when we are reopening the RocksDB. This causes ineffective file reuse and we end up not reusing files that should have been reused. Leading to a lot of unnecessary file upload/download, and in the worst case it will make us act like file reuse is disabled.
> # Ineffective file reuse when creating a checkpoint. We currently will not reuse the files for creating checkpoint, if it was added in the current version i.e. if you do load(v1) -> save(v2), the SST files loaded in v1 will not be reused in v2, and we will upload them again.
> These two bugs were not caught even though we have tests to catch them, because there was also a bug in the test.
> NOTE: these bugs will not cause corruption and more of a performance bug. They just make file reuse ineffective. And end up working like there's no file reuse enabled. i.e. A lot of file upload/downloadÂ
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org