(spark) branch branch-4.x updated: [SPARK-58925][SS][DOC] Document maxOpenFiles troubleshooting guidance

[ka...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

HeartSaVioR pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 0ddf746b9adc [SPARK-58925][SS][DOC] Document maxOpenFiles troubleshooting guidance
0ddf746b9adc is described below

commit 0ddf746b9adc8ca716b9781b780194cb313518d9
Author: Livia Zhu <li...@databricks.com>
AuthorDate: Fri Aug 21 12:29:06 2026 +0900

    [SPARK-58925][SS][DOC] Document maxOpenFiles troubleshooting guidance
    
    ### What changes were proposed in this pull request?
    
    Document that users should set
    `spark.sql.streaming.stateStore.rocksdb.maxOpenFiles` to a positive value when a
    streaming query fails to open RocksDB with `Too many open files`.
    
    JIRA: https://issues.apache.org/jira/browse/SPARK-58925
    
    ### Why are the changes needed?
    
    The existing documentation explains the open-file cache behavior but does not
    tell users how to configure RocksDB after reaching the operating system's open
    file limit.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a documentation-only update.
    
    ### How was this patch tested?
    
    Validated with `git diff --check`. No runtime tests are applicable to this
    documentation-only change.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-5)
    
    Closes #58183 from liviazhu/SPARK-58925-rocksdb-max-open-files-doc.
    
    Authored-by: Livia Zhu <li...@databricks.com>
    Signed-off-by: Jungtaek Lim <ka...@gmail.com>
    (cherry picked from commit ebdf5543b970c5c563037795ea66e84e44ad9588)
    Signed-off-by: Jungtaek Lim <ka...@gmail.com>
---
 docs/streaming/apis-on-dataframes-and-datasets.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/streaming/apis-on-dataframes-and-datasets.md b/docs/streaming/apis-on-dataframes-and-datasets.md
index 86585caead51..e946a8e38b89 100644
--- a/docs/streaming/apis-on-dataframes-and-datasets.md
+++ b/docs/streaming/apis-on-dataframes-and-datasets.md
@@ -1851,7 +1851,7 @@ Here are the configs regarding to RocksDB instance of the state store provider:
   </tr>
   <tr>
     <td>spark.sql.streaming.stateStore.rocksdb.maxOpenFiles</td>
-    <td>The number of open files that can be used by the RocksDB instance. Value of -1 means that files opened are always kept open. If the open file limit is reached, RocksDB will evict entries from the open file cache and close those file descriptors and remove the entries from the cache.</td>
+    <td>The number of open files that can be used by the RocksDB instance. Value of -1 means that files opened are always kept open. If the open file limit is reached, RocksDB will evict entries from the open file cache and close those file descriptors and remove the entries from the cache. Set this configuration to a positive value if a query fails when opening RocksDB with <code>Too many open files</code>.</td>
     <td>-1</td>
   </tr>
   <tr>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org