You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by nk...@apache.org on 2019/11/13 14:20:35 UTC
[flink] branch release-1.9 updated (aaae2d5 -> 359fa5a)
This is an automated email from the ASF dual-hosted git repository.
nkruber pushed a change to branch release-1.9
in repository https://gitbox.apache.org/repos/asf/flink.git.
from aaae2d5 [FLINK-14673][hive] Shouldn't expect HMS client to throw NoSuchObjectException for non-existing function
new 3b488a4 [hotfix][docs] clarify RocksDB thread options applicability per operator/TM
new 359fa5a [hotfix][docs] clarify that a RocksDB dependency in pom.xml may not be needed
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/_includes/generated/rocks_db_configurable_configuration.html | 4 ++--
docs/_includes/generated/rocks_db_configuration.html | 2 +-
docs/ops/state/state_backends.md | 7 ++++++-
.../flink/contrib/streaming/state/RocksDBConfigurableOptions.java | 4 ++--
.../org/apache/flink/contrib/streaming/state/RocksDBOptions.java | 2 +-
5 files changed, 12 insertions(+), 7 deletions(-)
[flink] 01/02: [hotfix][docs] clarify RocksDB thread options
applicability per operator/TM
Posted by nk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nkruber pushed a commit to branch release-1.9
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 3b488a4fdf2eb8307863aba0e6d4ba58e163b6ce
Author: Nico Kruber <ni...@ververica.com>
AuthorDate: Tue Nov 12 15:07:20 2019 +0100
[hotfix][docs] clarify RocksDB thread options applicability per operator/TM
---
docs/_includes/generated/rocks_db_configurable_configuration.html | 4 ++--
docs/_includes/generated/rocks_db_configuration.html | 2 +-
.../flink/contrib/streaming/state/RocksDBConfigurableOptions.java | 4 ++--
.../java/org/apache/flink/contrib/streaming/state/RocksDBOptions.java | 2 +-
4 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/docs/_includes/generated/rocks_db_configurable_configuration.html b/docs/_includes/generated/rocks_db_configurable_configuration.html
index 638a3a5..fa0e0ff 100644
--- a/docs/_includes/generated/rocks_db_configurable_configuration.html
+++ b/docs/_includes/generated/rocks_db_configurable_configuration.html
@@ -40,12 +40,12 @@
<tr>
<td><h5>state.backend.rocksdb.files.open</h5></td>
<td style="word-wrap: break-word;">(none)</td>
- <td>The maximum number of open files that can be used by the DB, '-1' means no limit. RocksDB has default configuration as '5000'.</td>
+ <td>The maximum number of open files (per TaskManager) that can be used by the DB, '-1' means no limit. RocksDB has default configuration as '5000'.</td>
</tr>
<tr>
<td><h5>state.backend.rocksdb.thread.num</h5></td>
<td style="word-wrap: break-word;">(none)</td>
- <td>The maximum number of concurrent background flush and compaction jobs. RocksDB has default configuration as '1'.</td>
+ <td>The maximum number of concurrent background flush and compaction jobs (per TaskManager). RocksDB has default configuration as '1'.</td>
</tr>
<tr>
<td><h5>state.backend.rocksdb.writebuffer.count</h5></td>
diff --git a/docs/_includes/generated/rocks_db_configuration.html b/docs/_includes/generated/rocks_db_configuration.html
index 88e8c46..f9a6d95 100644
--- a/docs/_includes/generated/rocks_db_configuration.html
+++ b/docs/_includes/generated/rocks_db_configuration.html
@@ -10,7 +10,7 @@
<tr>
<td><h5>state.backend.rocksdb.checkpoint.transfer.thread.num</h5></td>
<td style="word-wrap: break-word;">1</td>
- <td>The number of threads used to transfer (download and upload) files in RocksDBStateBackend.</td>
+ <td>The number of threads (per stateful operator) used to transfer (download and upload) files in RocksDBStateBackend.</td>
</tr>
<tr>
<td><h5>state.backend.rocksdb.localdir</h5></td>
diff --git a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBConfigurableOptions.java b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBConfigurableOptions.java
index 089f08c..261e5f0 100644
--- a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBConfigurableOptions.java
+++ b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBConfigurableOptions.java
@@ -46,13 +46,13 @@ public class RocksDBConfigurableOptions implements Serializable {
public static final ConfigOption<String> MAX_BACKGROUND_THREADS =
key("state.backend.rocksdb.thread.num")
.noDefaultValue()
- .withDescription("The maximum number of concurrent background flush and compaction jobs. " +
+ .withDescription("The maximum number of concurrent background flush and compaction jobs (per TaskManager). " +
"RocksDB has default configuration as '1'.");
public static final ConfigOption<String> MAX_OPEN_FILES =
key("state.backend.rocksdb.files.open")
.noDefaultValue()
- .withDescription("The maximum number of open files that can be used by the DB, '-1' means no limit. " +
+ .withDescription("The maximum number of open files (per TaskManager) that can be used by the DB, '-1' means no limit. " +
"RocksDB has default configuration as '5000'.");
//--------------------------------------------------------------------------
diff --git a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBOptions.java b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBOptions.java
index 8fa497b..8ffd142 100644
--- a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBOptions.java
+++ b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBOptions.java
@@ -56,7 +56,7 @@ public class RocksDBOptions {
public static final ConfigOption<Integer> CHECKPOINT_TRANSFER_THREAD_NUM = ConfigOptions
.key("state.backend.rocksdb.checkpoint.transfer.thread.num")
.defaultValue(1)
- .withDescription("The number of threads used to transfer (download and upload) files in RocksDBStateBackend.");
+ .withDescription("The number of threads (per stateful operator) used to transfer (download and upload) files in RocksDBStateBackend.");
/** This determines if compaction filter to cleanup state with TTL is enabled. */
public static final ConfigOption<Boolean> TTL_COMPACT_FILTER_ENABLED = ConfigOptions
[flink] 02/02: [hotfix][docs] clarify that a RocksDB dependency in
pom.xml may not be needed
Posted by nk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nkruber pushed a commit to branch release-1.9
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 359fa5a390adbabe2627bfd6fcc380e55fa3f669
Author: Nico Kruber <ni...@ververica.com>
AuthorDate: Tue Nov 12 15:09:52 2019 +0100
[hotfix][docs] clarify that a RocksDB dependency in pom.xml may not be needed
---
docs/ops/state/state_backends.md | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/docs/ops/state/state_backends.md b/docs/ops/state/state_backends.md
index c167803..cd0a041 100644
--- a/docs/ops/state/state_backends.md
+++ b/docs/ops/state/state_backends.md
@@ -146,16 +146,21 @@ env.setStateBackend(new FsStateBackend("hdfs://namenode:40010/flink/checkpoints"
</div>
</div>
-If you want to use the `RocksDBStateBackend`, then you have to add the following dependency to your Flink project.
+If you want to use the `RocksDBStateBackend` in your IDE or configure it programmatically in your Flink job, you will have to add the following dependency to your Flink project.
{% highlight xml %}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb{{ site.scala_version_suffix }}</artifactId>
<version>{{ site.version }}</version>
+ <scope>provided</scope>
</dependency>
{% endhighlight %}
+<div class="alert alert-info" markdown="span">
+ <strong>Note:</strong> Since RocksDB is part of the default Flink distribution, you do not need this dependency if you are not using any RocksDB code in your job and configure the state backend via `state.backend` and further [checkpointing]({{ site.baseurl }}/ops/config.html#checkpointing) and [RocksDB-specific]({{ site.baseurl }}/ops/config.html#rocksdb-state-backend) parameters in your `flink-conf.yaml`.
+</div>
+
### Setting Default State Backend