You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Neil Ramaswamy (Jira)" <ji...@apache.org> on 2024/06/25 05:04:00 UTC

[jira] [Created] (SPARK-48707) RocksDB setLevelCompactionDynamicLevelBytes should be set to false

Neil Ramaswamy created SPARK-48707:
--------------------------------------

             Summary: RocksDB setLevelCompactionDynamicLevelBytes should be set to false
                 Key: SPARK-48707
                 URL: https://issues.apache.org/jira/browse/SPARK-48707
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 4.0.0
            Reporter: Neil Ramaswamy


In RocksDB 8.4.0, they changed the default of `level_compaction_dynamic_level_bytes` to true. This means that any query that runs with this enabled will have an L6 in the LSM tree, so if a user rolls back to a previous version of Spark that doesn't have RocksDB 8.4.0, then their L6 of the LSM tree will not be compacted. We should set this conf to be false so that this rolling back works in Spark.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org