You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by zjureel <gi...@git.apache.org> on 2017/06/28 04:46:51 UTC
[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...
GitHub user zjureel opened a pull request:
https://github.com/apache/flink/pull/4204
[FLINK-6522] Add ZooKeeper cleanup logic to ZooKeeperHaServices
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjureel/flink FLINK-6522
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4204.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4204
----
commit b816aa3a2a3d4d1377caccdcbeba91073e89af75
Author: zjureel <zj...@gmail.com>
Date: 2017-06-28T02:12:27Z
clean up all data which is stored by RetrievableStateStorageHelper
commit 03a2f39e051594cb07017ef2613eb7873b56d118
Author: zjureel <zj...@gmail.com>
Date: 2017-06-28T04:42:30Z
clean up all data which is stored in zookeeper
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4204#discussion_r125164958
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java ---
@@ -244,6 +244,7 @@ public static ZooKeeperLeaderElectionService createLeaderElectionService(
* @return {@link ZooKeeperSubmittedJobGraphStore} instance
* @throws Exception if the submitted job graph store cannot be created
*/
+ @Deprecated
--- End diff --
Can we remove this method? If not, then at least write why this method was deprecated using the `@deprecated` tag.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4204#discussion_r125164898
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperHaServices.java ---
@@ -113,6 +117,7 @@ public ZooKeeperHaServices(
this.runningJobsRegistry = new ZooKeeperRunningJobsRegistry(client, configuration);
this.blobStoreService = checkNotNull(blobStoreService);
+ this.stateStorage = null;
--- End diff --
let's eagerly initialize this field
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4204#discussion_r125164895
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperHaServices.java ---
@@ -102,6 +104,8 @@
/** Store for arbitrary blobs */
private final BlobStoreService blobStoreService;
+ private RetrievableStateStorageService<SubmittedJobGraph> stateStorage;
--- End diff --
should be `final`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4204#discussion_r125164937
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/zookeeper/filesystem/FileSystemStateStorageHelper.java ---
@@ -77,4 +83,18 @@ public FileSystemStateStorageHelper(Path rootPath, String prefix) throws IOExcep
private Path getNewFilePath() {
return new Path(rootPath, FileUtils.getRandomFilename(prefix));
}
+
+ @Override
+ public void closeAndCleanupAllData() {
+ try {
+ fs.delete(rootPath, true);
+ } catch (Exception e) {
+ LOG.error("Failed to clean up state storage directory.", e);
--- End diff --
I would not catch this exception
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKeeperHaSe...
Posted by zjureel <gi...@git.apache.org>.
Github user zjureel commented on the issue:
https://github.com/apache/flink/pull/4204
Hi @tillrohrmann , I have created this PR for issue [FLINK-6522.](https://issues.apache.org/jira/browse/FLINK-6522) Could you please have a look when you're free, thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4204#discussion_r125164998
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/zookeeper/filesystem/FileSystemStateStorageHelper.java ---
@@ -77,4 +83,18 @@ public FileSystemStateStorageHelper(Path rootPath, String prefix) throws IOExcep
private Path getNewFilePath() {
return new Path(rootPath, FileUtils.getRandomFilename(prefix));
}
+
+ @Override
+ public void closeAndCleanupAllData() {
+ try {
+ fs.delete(rootPath, true);
--- End diff --
We cannot simply delete the rootPath, because it's `HighAvailabilityOptions.HA_STORAGE_PATH`. Thus, we would also delete files from other components in this directory (e.g. the `BlobServer`). I think the `FileSystemStateStorageHelper` has to create a sub directory for the state to store. That directory, we can easily delete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKee...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4204#discussion_r125164914
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperHaServices.java ---
@@ -190,6 +198,18 @@ public void closeAndCleanupAllData() throws Exception {
exception = t;
}
+ try {
+ stateStorage.closeAndCleanupAllData();
+ } catch (Throwable t) {
+ exception = t;
--- End diff --
add as suppressed exception using `ExceptionUtils.firstOrSuppressed`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4204: [FLINK-6522] Add ZooKeeper cleanup logic to ZooKeeperHaSe...
Posted by zjureel <gi...@git.apache.org>.
Github user zjureel commented on the issue:
https://github.com/apache/flink/pull/4204
@tillrohrmann Thank you for your review. I use `prefix` as the name of sub directory, and add test case to `FileSystemStateStorageHelper#closeAndCleanupAllData`. Also I have fixed the problem you metioned, thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---