You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by greghogan <gi...@git.apache.org> on 2015/10/20 18:43:19 UTC
[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/1272
[FLINK-2834] Global round-robin for temporary directories
Multiple TaskManager filesystems can be used by configuring multiple temporary directories.
This patch changes the process of spilling files from a per-operator round-robin to a global
round-robin such that each directory is written to in turn across all operators, reducing
unbalanced I/O due to bunching.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink 2834_global_round_robin_for_temporary_directories
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1272.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1272
----
commit 4af4332bfdf63112e940f706a6b8fedc0b7cff5b
Author: Greg Hogan <co...@greghogan.com>
Date: 2015-10-20T14:47:58Z
[FLINK-2834] Global round-robin for temporary directories
Multiple TaskManager filesystems can be used by configuring multiple temporary directories.
This patch changes the process of spilling files from a per-operator round-robin to a global
round-robin such that each directory is written to in turn across all operators, reducing
unbalanced I/O due to bunching.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1272#issuecomment-149636019
Looks good to me, +1 to merge this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1272
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1272#discussion_r42524229
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/FileIOChannel.java ---
@@ -154,20 +155,25 @@ private static String randomString(Random random) {
public static final class Enumerator {
private final File[] paths;
-
+
private final String namePrefix;
- private int counter;
+ private int localCounter;
+
+ private static AtomicInteger globalCounter = new AtomicInteger();
--- End diff --
Minor (optional) comment: In the (undocumented) code style that Flink follows implicitly, the static member fields are the first in the class, before the instance member fields.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1272#issuecomment-149838435
Will merge this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2834] Global round-robin for temporary ...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/1272#discussion_r42526400
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/iomanager/FileIOChannel.java ---
@@ -154,20 +155,25 @@ private static String randomString(Random random) {
public static final class Enumerator {
private final File[] paths;
-
+
private final String namePrefix;
- private int counter;
+ private int localCounter;
+
+ private static AtomicInteger globalCounter = new AtomicInteger();
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---