You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2014/09/25 19:19:53 UTC
[GitHub] incubator-flink pull request: Integraion of Daniel's BLOB manager
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/incubator-flink/pull/126
Integraion of Daniel's BLOB manager
I rebased the pull request #107 of @warneke. Thanks for your work Daniel, the code looks really good. Since the master evolved a little bit from the version of the pull request, I'd like to have another pair of eyes looking over it. I tried to address the problem of old jar files which are no longer used by introducing a clean up task which regularly looks for those files. Due to this I had to extend the book keeping of the LibraryCacheManager a little bit.
I tested WordCount and Kmeans on the cluster with this version.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/incubator-flink blob_manager
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/126.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #126
----
commit 68474d2ac589c81332d79fb03fc0c258a4b3d20c
Author: Daniel Warneke <wa...@apache.org>
Date: 2014-09-24T13:00:58Z
Integration of new BLOB service.
commit 80c023ac84cb91e6d53147c530b19736b1045e6c
Author: Till Rohrmann <tr...@apache.org>
Date: 2014-09-24T15:47:26Z
Looped through the user code class loader from the envrionment to the tasks.
commit defd8b9d394a77c5b49f6f0f2ccdcbe8fe5f9778
Author: Till Rohrmann <tr...@apache.org>
Date: 2014-09-25T16:41:37Z
Added timed cleanup task to LibraryCacheManager to remove automatically unreferenced jar files. Moved registration logic out of the IOReadableWritable serialization logic. Updated logger.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18420572
--- Diff: flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/util/ClusterUtil.java ---
@@ -37,18 +37,19 @@
*
* @param jobGraph
* jobGraph
- * @param numberOfTaskTrackers
+ * @param degreeOfPrallelism
* numberOfTaskTrackers
* @param memorySize
* memorySize
*/
- public static void runOnMiniCluster(JobGraph jobGraph, int numberOfTaskTrackers, long memorySize) throws Exception {
--- End diff --
Without this change, the FlinkMiniCluster does not wait properly on the TaskManagers which have to register at the JobManager. The reason for this is that the number of slots to wait for is negative. As a consequence, some of the test cases failed every once in a while.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18402505
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/JobManager.java ---
@@ -111,7 +112,8 @@
/** Executor service for asynchronous commands (to relieve the RPC threads of work) */
- private final ExecutorService executorService = Executors.newFixedThreadPool(2 * Hardware.getNumberCPUCores(), ExecutorThreadFactory.INSTANCE);
+ private final ExecutorService executorService = Executors.newFixedThreadPool(3 * Hardware
--- End diff --
Why change it from 2*cores to 3*cores?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18420375
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobKey.java ---
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.flink.runtime.blob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.MessageDigest;
+import java.util.Arrays;
+
+import org.apache.flink.core.io.IOReadableWritable;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+
+/**
+ * A BLOB key uniquely identifies a BLOB.
+ */
+public final class BlobKey implements IOReadableWritable, Comparable<BlobKey> {
+
+ /**
+ * Array of hex characters to facilitate fast toString() method.
+ */
+ private static final char[] HEX_ARRAY = "0123456789abcdef".toCharArray();
+
+ /**
+ * Size of the internal BLOB key in bytes.
+ */
+ private static final int SIZE = 20;
+
+ /**
+ * The byte buffer storing the actual key data.
+ */
+ private final byte[] key;
+
+ /**
+ * Constructs a new BLOB key.
+ */
+ public BlobKey() {
+ this.key = new byte[SIZE];
+ }
+
+ /**
+ * Constructs a new BLOB key from the given byte array.
+ *
+ * @param key
+ * the actual key data
+ */
+ BlobKey(final byte[] key) {
+
+ if (key.length != SIZE) {
+ throw new IllegalArgumentException("BLOB key must have a size of " + SIZE + " bytes");
+ }
+
+ this.key = key;
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void write(final DataOutputView out) throws IOException {
+ out.write(this.key);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void read(final DataInputView in) throws IOException {
+ in.readFully(this.key);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public boolean equals(final Object obj) {
+
+ if (!(obj instanceof BlobKey)) {
+ return false;
+ }
+
+ final BlobKey bk = (BlobKey) obj;
+
+ return Arrays.equals(this.key, bk.key);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public int hashCode() {
+ return Arrays.hashCode(this.key);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public String toString() {
+
+ final char[] hexChars = new char[SIZE * 2];
--- End diff --
Good point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18402567
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobKey.java ---
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.flink.runtime.blob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.security.MessageDigest;
+import java.util.Arrays;
+
+import org.apache.flink.core.io.IOReadableWritable;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+
+/**
+ * A BLOB key uniquely identifies a BLOB.
+ */
+public final class BlobKey implements IOReadableWritable, Comparable<BlobKey> {
+
+ /**
+ * Array of hex characters to facilitate fast toString() method.
+ */
+ private static final char[] HEX_ARRAY = "0123456789abcdef".toCharArray();
+
+ /**
+ * Size of the internal BLOB key in bytes.
+ */
+ private static final int SIZE = 20;
+
+ /**
+ * The byte buffer storing the actual key data.
+ */
+ private final byte[] key;
+
+ /**
+ * Constructs a new BLOB key.
+ */
+ public BlobKey() {
+ this.key = new byte[SIZE];
+ }
+
+ /**
+ * Constructs a new BLOB key from the given byte array.
+ *
+ * @param key
+ * the actual key data
+ */
+ BlobKey(final byte[] key) {
+
+ if (key.length != SIZE) {
+ throw new IllegalArgumentException("BLOB key must have a size of " + SIZE + " bytes");
+ }
+
+ this.key = key;
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void write(final DataOutputView out) throws IOException {
+ out.write(this.key);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public void read(final DataInputView in) throws IOException {
+ in.readFully(this.key);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public boolean equals(final Object obj) {
+
+ if (!(obj instanceof BlobKey)) {
+ return false;
+ }
+
+ final BlobKey bk = (BlobKey) obj;
+
+ return Arrays.equals(this.key, bk.key);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public int hashCode() {
+ return Arrays.hashCode(this.key);
+ }
+
+ /**
+ * {@inheritDoc}
+ */
+ @Override
+ public String toString() {
+
+ final char[] hexChars = new char[SIZE * 2];
--- End diff --
This seems to be a solution taken from stackoverflow. I think the code should attribute that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18402388
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/JobManager.java ---
@@ -309,41 +326,31 @@ public JobSubmissionResult submitJob(JobGraph job) throws IOException {
LOG.info(String.format("Found existing execution graph for id %s, attaching this job.", job.getJobID()));
}
}
+
+ // Register for updates on the job status
+ executionGraph.registerJobStatusListener(this);
// grab the class loader for user-defined code
- final ClassLoader userCodeLoader = LibraryCacheManager.getClassLoader(job.getJobID());
+ final ClassLoader userCodeLoader = libraryCacheManager.getClassLoader(job.getJobID());
if (userCodeLoader == null) {
throw new JobException("The user code class loader could not be initialized.");
}
-
- String[] jarFilesForJob = LibraryCacheManager.getRequiredJarFiles(job.getJobID());
- for (String fileId : jarFilesForJob) {
- executionGraph.addUserCodeJarFile(fileId);
- }
-
+
// first, perform the master initialization of the nodes
if (LOG.isDebugEnabled()) {
LOG.debug(String.format("Running master initialization of job %s (%s)", job.getJobID(), job.getName()));
}
- try {
- for (AbstractJobVertex vertex : job.getVertices()) {
- // check that the vertex has an executable class
- String executableClass = vertex.getInvokableClassName();
- if (executableClass == null || executableClass.length() == 0) {
- throw new JobException(String.format("The vertex %s (%s) has no invokable class.", vertex.getID(), vertex.getName()));
- }
-
- // master side initialization
- vertex.initializeOnMaster(userCodeLoader);
+ for (AbstractJobVertex vertex : job.getVertices()) {
+ // check that the vertex has an executable class
+ String executableClass = vertex.getInvokableClassName();
+ if (executableClass == null || executableClass.length() == 0) {
+ throw new JobException(String.format("The vertex %s (%s) has no invokable class.", vertex.getID(), vertex.getName()));
}
+
+ // master side initialization
+ vertex.initializeOnMaster(userCodeLoader);
}
- catch (FileNotFoundException e) {
--- End diff --
This was purposefully added to get a better (simpler) error message in the most common failure case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18436668
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/JobManager.java ---
@@ -309,41 +326,31 @@ public JobSubmissionResult submitJob(JobGraph job) throws IOException {
LOG.info(String.format("Found existing execution graph for id %s, attaching this job.", job.getJobID()));
}
}
+
+ // Register for updates on the job status
+ executionGraph.registerJobStatusListener(this);
// grab the class loader for user-defined code
- final ClassLoader userCodeLoader = LibraryCacheManager.getClassLoader(job.getJobID());
+ final ClassLoader userCodeLoader = libraryCacheManager.getClassLoader(job.getJobID());
if (userCodeLoader == null) {
throw new JobException("The user code class loader could not be initialized.");
}
-
- String[] jarFilesForJob = LibraryCacheManager.getRequiredJarFiles(job.getJobID());
- for (String fileId : jarFilesForJob) {
- executionGraph.addUserCodeJarFile(fileId);
- }
-
+
// first, perform the master initialization of the nodes
if (LOG.isDebugEnabled()) {
LOG.debug(String.format("Running master initialization of job %s (%s)", job.getJobID(), job.getName()));
}
- try {
- for (AbstractJobVertex vertex : job.getVertices()) {
- // check that the vertex has an executable class
- String executableClass = vertex.getInvokableClassName();
- if (executableClass == null || executableClass.length() == 0) {
- throw new JobException(String.format("The vertex %s (%s) has no invokable class.", vertex.getID(), vertex.getName()));
- }
-
- // master side initialization
- vertex.initializeOnMaster(userCodeLoader);
+ for (AbstractJobVertex vertex : job.getVertices()) {
+ // check that the vertex has an executable class
+ String executableClass = vertex.getInvokableClassName();
+ if (executableClass == null || executableClass.length() == 0) {
+ throw new JobException(String.format("The vertex %s (%s) has no invokable class.", vertex.getID(), vertex.getName()));
}
+
+ // master side initialization
+ vertex.initializeOnMaster(userCodeLoader);
}
- catch (FileNotFoundException e) {
--- End diff --
I think this was an exception from java, where we had no influence on the error message.
Any way that gives a good error message is fine with me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18400363
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/iterative/task/IterationHeadPactTask.java ---
@@ -157,6 +158,7 @@ private BlockingBackChannel initBackChannel() throws Exception {
private <BT> CompactingHashTable<BT> initCompactingHashTable() throws Exception {
// get some memory
double hashjoinMemorySize = config.getRelativeSolutionSetMemory();
+ final ClassLoader userCodeClassLoader = getEnvironment().getUserClassLoader();
--- End diff --
At all these points, we already have the user code class loader in a protected variable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18399289
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCache.java ---
@@ -0,0 +1,156 @@
+/**
--- End diff --
Still has the javadoc license headers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18420335
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/JobManager.java ---
@@ -111,7 +112,8 @@
/** Executor service for asynchronous commands (to relieve the RPC threads of work) */
- private final ExecutorService executorService = Executors.newFixedThreadPool(2 * Hardware.getNumberCPUCores(), ExecutorThreadFactory.INSTANCE);
+ private final ExecutorService executorService = Executors.newFixedThreadPool(3 * Hardware
--- End diff --
That is just a debugging relict. I'll change it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18420291
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/JobManager.java ---
@@ -309,41 +326,31 @@ public JobSubmissionResult submitJob(JobGraph job) throws IOException {
LOG.info(String.format("Found existing execution graph for id %s, attaching this job.", job.getJobID()));
}
}
+
+ // Register for updates on the job status
+ executionGraph.registerJobStatusListener(this);
// grab the class loader for user-defined code
- final ClassLoader userCodeLoader = LibraryCacheManager.getClassLoader(job.getJobID());
+ final ClassLoader userCodeLoader = libraryCacheManager.getClassLoader(job.getJobID());
if (userCodeLoader == null) {
throw new JobException("The user code class loader could not be initialized.");
}
-
- String[] jarFilesForJob = LibraryCacheManager.getRequiredJarFiles(job.getJobID());
- for (String fileId : jarFilesForJob) {
- executionGraph.addUserCodeJarFile(fileId);
- }
-
+
// first, perform the master initialization of the nodes
if (LOG.isDebugEnabled()) {
LOG.debug(String.format("Running master initialization of job %s (%s)", job.getJobID(), job.getName()));
}
- try {
- for (AbstractJobVertex vertex : job.getVertices()) {
- // check that the vertex has an executable class
- String executableClass = vertex.getInvokableClassName();
- if (executableClass == null || executableClass.length() == 0) {
- throw new JobException(String.format("The vertex %s (%s) has no invokable class.", vertex.getID(), vertex.getName()));
- }
-
- // master side initialization
- vertex.initializeOnMaster(userCodeLoader);
+ for (AbstractJobVertex vertex : job.getVertices()) {
+ // check that the vertex has an executable class
+ String executableClass = vertex.getInvokableClassName();
+ if (executableClass == null || executableClass.length() == 0) {
+ throw new JobException(String.format("The vertex %s (%s) has no invokable class.", vertex.getID(), vertex.getName()));
}
+
+ // master side initialization
+ vertex.initializeOnMaster(userCodeLoader);
}
- catch (FileNotFoundException e) {
--- End diff --
I reinsert the more detailed exception. Initially, I just removed it because of the return statement which disrupts the control flow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/incubator-flink/pull/126#issuecomment-57151716
@StephanEwen, I updated the LibraryCacheManager so that it is no longer a singleton.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/incubator-flink/pull/126#issuecomment-57950074
I addressed the comments. The BlobLibraryCacheManager now uses a global lock to synchronize the function calls.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18402745
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java ---
@@ -0,0 +1,345 @@
+/*
--- End diff --
I think the logic with the lock map is dangerous, and most likely even not more efficient than a simple lock.
Previous versions of the LibraryCacheManager has errors that the locks were not always removed, leading to endless spins. Manual spinning is not required any more, the JVM does adaptive spinning that automatically degrades to queued waiting, which is better than anything we could hope to implement manually.
Since we do not expect congestion on this class, even the most simple global lock might do (and would be lightweight for the JVM).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18421462
--- Diff: flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/util/ClusterUtil.java ---
@@ -37,18 +37,19 @@
*
* @param jobGraph
* jobGraph
- * @param numberOfTaskTrackers
+ * @param degreeOfPrallelism
* numberOfTaskTrackers
* @param memorySize
* memorySize
*/
- public static void runOnMiniCluster(JobGraph jobGraph, int numberOfTaskTrackers, long memorySize) throws Exception {
--- End diff --
Ah, makes sense. I guess it is also a good idea to have only one task manager by default.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-flink/pull/126
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18436643
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/JobManager.java ---
@@ -309,41 +326,31 @@ public JobSubmissionResult submitJob(JobGraph job) throws IOException {
LOG.info(String.format("Found existing execution graph for id %s, attaching this job.", job.getJobID()));
}
}
+
+ // Register for updates on the job status
+ executionGraph.registerJobStatusListener(this);
// grab the class loader for user-defined code
- final ClassLoader userCodeLoader = LibraryCacheManager.getClassLoader(job.getJobID());
+ final ClassLoader userCodeLoader = libraryCacheManager.getClassLoader(job.getJobID());
if (userCodeLoader == null) {
throw new JobException("The user code class loader could not be initialized.");
}
-
- String[] jarFilesForJob = LibraryCacheManager.getRequiredJarFiles(job.getJobID());
- for (String fileId : jarFilesForJob) {
- executionGraph.addUserCodeJarFile(fileId);
- }
-
+
// first, perform the master initialization of the nodes
if (LOG.isDebugEnabled()) {
LOG.debug(String.format("Running master initialization of job %s (%s)", job.getJobID(), job.getName()));
}
- try {
- for (AbstractJobVertex vertex : job.getVertices()) {
- // check that the vertex has an executable class
- String executableClass = vertex.getInvokableClassName();
- if (executableClass == null || executableClass.length() == 0) {
- throw new JobException(String.format("The vertex %s (%s) has no invokable class.", vertex.getID(), vertex.getName()));
- }
-
- // master side initialization
- vertex.initializeOnMaster(userCodeLoader);
+ for (AbstractJobVertex vertex : job.getVertices()) {
+ // check that the vertex has an executable class
+ String executableClass = vertex.getInvokableClassName();
+ if (executableClass == null || executableClass.length() == 0) {
+ throw new JobException(String.format("The vertex %s (%s) has no invokable class.", vertex.getID(), vertex.getName()));
}
+
+ // master side initialization
+ vertex.initializeOnMaster(userCodeLoader);
}
- catch (FileNotFoundException e) {
--- End diff --
But if the error message is too misleading, shouldn't it be the responsibility of the function throwing the exception to generate a meaningful error message instead of the job manager's?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: Integration of Daniel's BLOB manager
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/126#discussion_r18402807
--- Diff: flink-addons/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/util/ClusterUtil.java ---
@@ -37,18 +37,19 @@
*
* @param jobGraph
* jobGraph
- * @param numberOfTaskTrackers
+ * @param degreeOfPrallelism
* numberOfTaskTrackers
* @param memorySize
* memorySize
*/
- public static void runOnMiniCluster(JobGraph jobGraph, int numberOfTaskTrackers, long memorySize) throws Exception {
--- End diff --
Is that really in the sense of what the streaming folks had in mind? Probably makes little difference when everything is in the same JVM, but in general, I think such edits should not be mixed into other pull requests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---