You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2014/10/17 15:52:53 UTC
[GitHub] incubator-flink pull request: [FLINK-1170] Fix faulty input split ...
GitHub user rmetzger opened a pull request:
https://github.com/apache/incubator-flink/pull/158
[FLINK-1170] Fix faulty input split localization
Pass hostname to split assigner
Avoid clashes by only using the first component of the fully qualified hostname
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/incubator-flink localizableInputSplitAssignment
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/158.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #158
----
commit 8896b091c2c707f4df331f7f40a89375e2f6f02d
Author: Robert Metzger <rm...@apache.org>
Date: 2014-10-17T08:58:15Z
[FLINK-1170] Fix faulty input split localization
Pass hostname to split assigner
Avoid clashes by only using the first component of the fully qualified hostname
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1170] Fix faulty input split ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/158#discussion_r19032342
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/instance/InstanceConnectionInfo.java ---
@@ -154,7 +176,7 @@ public void read(DataInputView in) throws IOException {
this.ipcPort = in.readInt();
this.dataPort = in.readInt();
- this.hostName = StringUtils.readNullableString(in);
--- End diff --
Would either serialize/deserialize the hostname as well, or at least set it to null.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1170] Fix faulty input split ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/158#discussion_r19032254
--- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/LocatableInputSplitAssigner.java ---
@@ -173,13 +175,13 @@ public LocatableInputSplit getNextInputSplit(String host) {
}
}
- private static final boolean isLocal(String host, String[] hosts) {
- if (host == null || hosts == null) {
+ private static final boolean isLocal(String flinkHost, String[] hosts) {
+ if (flinkHost == null || hosts == null) {
return false;
}
-
for (String h : hosts) {
--- End diff --
I would add some null checks here. Input Splits are user generated an can contained arbitrary nonsense in general
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1170] Fix faulty input split ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-flink/pull/158
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1170] Fix faulty input split ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/158#discussion_r19032889
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/instance/InstanceConnectionInfo.java ---
@@ -25,7 +25,10 @@
import org.apache.flink.core.io.IOReadableWritable;
--- End diff --
Eager computation of the FQDN (in the constructor) on the taskmanager would be a good idea. Otherwise, in corner cases, reverse name lookup falls to the jobmanager (which might get overloaded doing too many reverse lookups).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1170] Fix faulty input split ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/158#discussion_r19032266
--- Diff: flink-core/src/main/java/org/apache/flink/util/NetUtils.java ---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.util;
+
+
+public class NetUtils {
+ /**
+ * Turn a fully qualified domain name (fqdn) into a hostname.
+ *
+ * @param fqdn
+ * @return
+ */
+ public static String getHostnameFromFQDN(String fqdn) {
+ int dotPos = fqdn.indexOf('.');
--- End diff --
Would add a null check here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1170] Fix faulty input split ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/158#issuecomment-59552098
Other than the comments, I think this is good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---