You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by bowenli86 <gi...@git.apache.org> on 2017/06/18 07:19:08 UTC
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
GitHub user bowenli86 opened a pull request:
https://github.com/apache/flink/pull/4136
[FLINK-6940][docs] Clarify the effect of configuring per-job state backend
The documentation of having different options configuring flink state backend is confusing. We should add explicit doc explaining configuring a per-job flink state backend in code will overwrite any default state backend configured in flink-conf.yaml
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
-Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bowenli86/flink FLINK-6940
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4136.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4136
----
commit 6ef758bfc8ef44a1ed6061ce85af849cc3f65c96
Author: Bowen Li <bo...@gmail.com>
Date: 2017-06-18T07:17:20Z
[FLINK-6940][docs] Clarify the effect of configuring per-job state backend
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r126240877
--- Diff: docs/ops/state_backends.md ---
@@ -124,7 +124,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+job does not explicitly define a state backend. Besides, state backend configured per-job will overwrite the default state backend configured in `flink-conf.yaml`
--- End diff --
@zentol any suggestions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by alpinegizmo <gi...@git.apache.org>.
Github user alpinegizmo commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r126431179
--- Diff: docs/ops/state_backends.md ---
@@ -123,8 +123,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
-State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+State backends can be configured per job in code. In addition, you can define a default state backend in **flink-conf.yaml** that is used when the job does not explicitly define a state backend.
--- End diff --
@zentol I find that "in code" reads rather awkwardly, and I don't see how it adds any value, since the details of how to do per-job configuration are shown below. Nevertheless, this topic can be a bit confusing, so I would suggest something more like this (assuming I got the details right):
The default state backend, if you specify nothing, is the jobmanager. If you wish to establish a different default for all jobs on your cluster, you can do so by defining a new default state backend in **flink-conf.yaml**. The default state backend can be overridden on a per-job basis, as shown below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4136
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r126401914
--- Diff: docs/ops/state_backends.md ---
@@ -123,8 +123,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
-State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+State backends can be configured per job in code. In addition, you can define a default state backend in **flink-conf.yaml** that is used when the job does not explicitly define a state backend.
--- End diff --
huh...i just noticed the 2 subsections below that describe how to configure the default/per-job state backend. Aren't we just duplicating details here?
@alpinegizmo Do you have any input?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r123306543
--- Diff: docs/ops/state_backends.md ---
@@ -124,7 +124,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+job does not explicitly define a state backend. Besides, state backend configured per-job will overwrite the default state backend configured in `flink-conf.yaml`
--- End diff --
Yeah, this one works too.
How about "State backends can be configured per job in code. In addition, you can define a default state backend in flink-conf.yaml that is used when the job does not explicitly define a state backend."?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4136: [FLINK-6940][docs] Clarify the effect of configuring per-...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4136
@zentol @alpinegizmo Let me know your thoughts on it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r126275617
--- Diff: docs/ops/state_backends.md ---
@@ -124,7 +124,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+job does not explicitly define a state backend. Besides, state backend configured per-job will overwrite the default state backend configured in `flink-conf.yaml`
--- End diff --
sorry for the wait, I'll take a look at this PR on monday.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r123228173
--- Diff: docs/ops/state_backends.md ---
@@ -124,7 +124,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+job does not explicitly define a state backend. Besides, state backend configured per-job will overwrite the default state backend configured in `flink-conf.yaml`
--- End diff --
Would it be enough to reword the existing docs to:
"
State backends can be configured per job. In addition, you can define a default state backend **in `flink-conf.yaml`** to be used when the job does not explicitly define a state backend.
"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r126503269
--- Diff: docs/ops/state_backends.md ---
@@ -123,8 +123,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
-State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+State backends can be configured per job in code. In addition, you can define a default state backend in **flink-conf.yaml** that is used when the job does not explicitly define a state backend.
--- End diff --
This is probably more readable. I'll update doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r123089833
--- Diff: docs/ops/state_backends.md ---
@@ -124,7 +124,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+job does not explicitly define a state backend. Besides, state backend configured per-job will overwrite the default state backend configured in `flink-conf.yaml`
--- End diff --
I don't believe implying something is enough.
My teammates and I went through this piece of documentation several times when trying to enable checkpoints, but couldn't figure out the exact configurations and the relationship among several config/code keys. Until we shout out in user list http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/confusing-RocksDBStateBackend-parameters-td13810.html
Thus I believe the explicit explanation is absolutely necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4136: [FLINK-6940][docs] Clarify the effect of configuring per-...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4136
will merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4136: [FLINK-6940][docs] Clarify the effect of configuri...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4136#discussion_r122970989
--- Diff: docs/ops/state_backends.md ---
@@ -124,7 +124,7 @@ RocksDBStateBackend is currently the only backend that offers incremental checkp
## Configuring a State Backend
State backends can be configured per job. In addition, you can define a default state backend to be used when the
-job does not explicitly define a state backend.
+job does not explicitly define a state backend. Besides, state backend configured per-job will overwrite the default state backend configured in `flink-conf.yaml`
--- End diff --
this addition is redundant, as " you can define a default state backend to be used when the
job does not explicitly define a state backend." already implies this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4136: [FLINK-6940][docs] Clarify the effect of configuring per-...
Posted by bowenli86 <gi...@git.apache.org>.
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4136
@zentol @alpinegizmo Guys, please let me know your thoughts :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---