You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anishshri-db (via GitHub)" <gi...@apache.org> on 2023/05/04 01:01:34 UTC
[GitHub] [spark] anishshri-db opened a new pull request, #41042: [SPARK-43364] Add docs for RocksDB state store memory management
anishshri-db opened a new pull request, #41042:
URL: https://github.com/apache/spark/pull/41042
### What changes were proposed in this pull request?
Add docs for RocksDB state store memory management
### Why are the changes needed?
Docs only change
### Does this PR introduce _any_ user-facing change?
N/A
### How was this patch tested?
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on pull request #41042: [SPARK-43364] Add docs for RocksDB state store memory management
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on PR #41042:
URL: https://github.com/apache/spark/pull/41042#issuecomment-1533931491
@HeartSaVioR @siying - please take a look. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] siying commented on a diff in pull request #41042: [SPARK-43364][SS] Add docs for RocksDB state store memory management
Posted by "siying (via GitHub)" <gi...@apache.org>.
siying commented on code in PR #41042:
URL: https://github.com/apache/spark/pull/41042#discussion_r1185399331
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
+ <td>Whether total memory usage for RocksDB state store instances on a single node is bounded.</td>
+ <td>false</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB</td>
+ <td>Total memory limit in MB for RocksDB state store instances on a single node.</td>
+ <td>500</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.writeBufferCacheRatio</td>
+ <td>Total memory to be occupied by write buffers as a fraction of memory allocated across all RocksDB instances on a single node.</td>
Review Comment:
Perhaps be more specific that the base is `maxMemoryUsageMB`, although it can be obvious to some users.
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
+ <td>Whether total memory usage for RocksDB state store instances on a single node is bounded.</td>
+ <td>false</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB</td>
+ <td>Total memory limit in MB for RocksDB state store instances on a single node.</td>
+ <td>500</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.writeBufferCacheRatio</td>
+ <td>Total memory to be occupied by write buffers as a fraction of memory allocated across all RocksDB instances on a single node.</td>
+ <td>0.5</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.highPriorityPoolRatio</td>
+ <td>Total memory to be occupied by filter and index blocks as a fraction of memory allocated across all RocksDB instances on a single node.</td>
+ <td>0.1</td>
Review Comment:
I didn't realize the default is 0.1. If we don't have a special study, 0.5 is a good value to start with.
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
+ <td>Whether total memory usage for RocksDB state store instances on a single node is bounded.</td>
+ <td>false</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB</td>
+ <td>Total memory limit in MB for RocksDB state store instances on a single node.</td>
+ <td>500</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.writeBufferCacheRatio</td>
+ <td>Total memory to be occupied by write buffers as a fraction of memory allocated across all RocksDB instances on a single node.</td>
+ <td>0.5</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.highPriorityPoolRatio</td>
+ <td>Total memory to be occupied by filter and index blocks as a fraction of memory allocated across all RocksDB instances on a single node.</td>
Review Comment:
It is more than filter and index blocks. It's the high pri pool size, used for mid-point insertion. Blocks are first inserted to low pri pool and promote to high pri pool for the second access. Filter and index blocks can directly go to high pri pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #41042: [SPARK-43364][SS] Add docs for RocksDB state store memory management
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #41042:
URL: https://github.com/apache/spark/pull/41042#discussion_r1186958114
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
Review Comment:
Maybe better to defer this till we make RocksDB state store provider as default implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41042: [SPARK-43364][SS] Add docs for RocksDB state store memory management
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41042:
URL: https://github.com/apache/spark/pull/41042#discussion_r1186955199
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
Review Comment:
Wonder if we can add them into `SQLConf` or `StaticSQLConf` so the docs can be generated, and to make it clear which value is the default.
For instance, we automatically generate the docs from SQL conf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #41042: [SPARK-43364][SS][DOCS] Add docs for RocksDB state store memory management
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #41042: [SPARK-43364][SS][DOCS] Add docs for RocksDB state store memory management
URL: https://github.com/apache/spark/pull/41042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #41042: [SPARK-43364][SS] Add docs for RocksDB state store memory management
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41042:
URL: https://github.com/apache/spark/pull/41042#issuecomment-1537616320
cc @HeartSaVioR FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #41042: [SPARK-43364][SS] Add docs for RocksDB state store memory management
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #41042:
URL: https://github.com/apache/spark/pull/41042#discussion_r1185419718
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
+ <td>Whether total memory usage for RocksDB state store instances on a single node is bounded.</td>
+ <td>false</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB</td>
+ <td>Total memory limit in MB for RocksDB state store instances on a single node.</td>
+ <td>500</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.writeBufferCacheRatio</td>
+ <td>Total memory to be occupied by write buffers as a fraction of memory allocated across all RocksDB instances on a single node.</td>
+ <td>0.5</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.highPriorityPoolRatio</td>
+ <td>Total memory to be occupied by filter and index blocks as a fraction of memory allocated across all RocksDB instances on a single node.</td>
+ <td>0.1</td>
Review Comment:
Yes, this is for the high pri pool blocks. The default for write buffer ratio is 0.5. Do we not need to keep some fraction for elements in the block cache too ?
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
+ <td>Whether total memory usage for RocksDB state store instances on a single node is bounded.</td>
+ <td>false</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB</td>
+ <td>Total memory limit in MB for RocksDB state store instances on a single node.</td>
+ <td>500</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.writeBufferCacheRatio</td>
+ <td>Total memory to be occupied by write buffers as a fraction of memory allocated across all RocksDB instances on a single node.</td>
+ <td>0.5</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.highPriorityPoolRatio</td>
+ <td>Total memory to be occupied by filter and index blocks as a fraction of memory allocated across all RocksDB instances on a single node.</td>
+ <td>0.1</td>
Review Comment:
Yes, this is for the high pri pool blocks. The default for write buffer ratio is 0.5. Do we not need to keep some fraction for elements in the regular block cache too ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #41042: [SPARK-43364][SS] Add docs for RocksDB state store memory management
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #41042:
URL: https://github.com/apache/spark/pull/41042#issuecomment-1537621497
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] anishshri-db commented on a diff in pull request #41042: [SPARK-43364][SS] Add docs for RocksDB state store memory management
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #41042:
URL: https://github.com/apache/spark/pull/41042#discussion_r1185418799
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
+ <td>Whether total memory usage for RocksDB state store instances on a single node is bounded.</td>
+ <td>false</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB</td>
+ <td>Total memory limit in MB for RocksDB state store instances on a single node.</td>
+ <td>500</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.writeBufferCacheRatio</td>
+ <td>Total memory to be occupied by write buffers as a fraction of memory allocated across all RocksDB instances on a single node.</td>
Review Comment:
Done
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
+ <td>Whether total memory usage for RocksDB state store instances on a single node is bounded.</td>
+ <td>false</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB</td>
+ <td>Total memory limit in MB for RocksDB state store instances on a single node.</td>
+ <td>500</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.writeBufferCacheRatio</td>
+ <td>Total memory to be occupied by write buffers as a fraction of memory allocated across all RocksDB instances on a single node.</td>
+ <td>0.5</td>
+ </tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.highPriorityPoolRatio</td>
+ <td>Total memory to be occupied by filter and index blocks as a fraction of memory allocated across all RocksDB instances on a single node.</td>
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41042: [SPARK-43364][SS] Add docs for RocksDB state store memory management
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41042:
URL: https://github.com/apache/spark/pull/41042#discussion_r1186955261
##########
docs/structured-streaming-programming-guide.md:
##########
@@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider:
<td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td>
<td>-1</td>
</tr>
+ <tr>
+ <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td>
Review Comment:
Can be done separately in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org