You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by stefanobortoli <gi...@git.apache.org> on 2016/04/19 18:39:13 UTC
[GitHub] flink pull request: Flink 1827 and small fixes in some tests
GitHub user stefanobortoli opened a pull request:
https://github.com/apache/flink/pull/1915
Flink 1827 and small fixes in some tests
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [X] General
- The pull request references the related JIRA issue
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message
- [X] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [X] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/stefanobortoli/flink FLINK-1827
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1915.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1915
----
commit 88483205c57884303c9e46159df9d89aa953b547
Author: okkam <ok...@okkamvm>
Date: 2016-04-19T09:50:59Z
FLINK-1827 fixed compilation that skips tests
commit 5a5cc98ba4fc407fe0fc18d7a57dc4455ce80001
Author: okkam <ok...@okkamvm>
Date: 2016-04-19T09:50:59Z
FLINK-1827 fixed compilation that skips tests
commit 85b1763ea84a59258a3a4fef87d05650e99b5be0
Author: okkam <ok...@okkamvm>
Date: 2016-04-19T09:58:13Z
Merge remote-tracking branch 'origin/FLINK-1827' into FLINK-1827
Conflicts:
flink-test-utils/pom.xml
commit 310b4e901283fde317dc97c9d454341d331f5d04
Author: okkam <ok...@okkamvm>
Date: 2016-04-19T16:37:55Z
FLINK-1827 and improved tests removing references to localized messages
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by stefanobortoli <gi...@git.apache.org>.
Github user stefanobortoli commented on a diff in the pull request:
https://github.com/apache/flink/pull/1915#discussion_r61037816
--- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/taskmanager/TaskManagerStartupTest.java ---
@@ -53,41 +59,55 @@ public void testStartupWhenTaskmanagerActorPortIsUsed() {
blocker = new ServerSocket(0, 50, localAddress);
final int port = blocker.getLocalPort();
- try {
- TaskManager.runTaskManager(
- localHostName,
- ResourceID.generate(),
- port,
- new Configuration(),
+ TaskManager.runTaskManager(localHostName, ResourceID.generate(), port, new Configuration(),
TaskManager.class);
- fail("This should fail with an IOException");
- }
- catch (IOException e) {
- // expected. validate the error message
- assertNotNull(e.getMessage());
- assertTrue(e.getMessage().contains("Address already in use"));
+ fail("This should fail with an IOException");
+
+ } catch (IOException e) {
+ // expected. validate the error messagex
+ List<Throwable> causes = getExceptionCauses(e, new ArrayList<Throwable>());
+ for (Throwable cause : causes) {
+ if (cause instanceof BindException) {
+ throw (BindException) cause;
+ }
}
-
- }
- catch (Exception e) {
+ fail("This should fail with an exception caused by BindException");
+ } catch (Exception e) {
e.printStackTrace();
fail(e.getMessage());
- }
- finally {
+ } finally {
if (blocker != null) {
try {
blocker.close();
- }
- catch (IOException e) {
+ } catch (IOException e) {
// no need to log here
}
}
}
}
/**
- * Tests that the TaskManager startup fails synchronously when the I/O directories are
- * not writable.
+ * A utility method to analyze the exceptions and collect the clauses
+ *
+ * @param e
+ * the root exception (Throwable) object
+ * @param causes
+ * the list of exceptions that caused the root exceptions
+ * @return
+ */
+ private List<Throwable> getExceptionCauses(Throwable e, List<Throwable> causes) {
--- End diff --
sure I can do that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by stefanobortoli <gi...@git.apache.org>.
Github user stefanobortoli commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-216521677
@tillrohrmann, I see the conflicts. How should I deal with this? rebase?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Flink 1827 and small fixes in some tests
Posted by fpompermaier <gi...@git.apache.org>.
Github user fpompermaier commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-212390687
With this last commit we improved test stability. With respect to other PR (that fails in all configuration) this PR fails only in the last one, when compiling **flink-ml**. However, this is an error that happens all the times (looking at other PRs) so it should be definitely fixed IMHO.
The error is:
```
--- maven-scala-plugin:2.15.2:compile (default) @ flink-ml_2.10 ---
[INFO] Checking for multiple versions of scala
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[INFO] /home/travis/build/apache/flink/flink-libraries/flink-ml/src/main/scala:-1: info: compiling
[INFO] Compiling 43 source files to /home/travis/build/apache/flink/flink-libraries/flink-ml/target/classes at 1461086334646
[ERROR] error: error while loading <root>, error in opening zip file
[ERROR] error: scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found.
[ERROR] at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
[ERROR] at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:172)
[INFO] at scala.reflect.internal.Mirrors$RootsBase.getRequiredPackage(Mirrors.scala:175)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage$lzycompute(Definitions.scala:183)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage(Definitions.scala:183)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass$lzycompute(Definitions.scala:184)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass(Definitions.scala:184)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr$lzycompute(Definitions.scala:1024)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr(Definitions.scala:1023)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1153)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1152)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1196)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1196)
[INFO] at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1261)
[INFO] at scala.tools.nsc.Global$Run.<init>(Global.scala:1290)
[INFO] at scala.tools.nsc.Driver.doCompile(Driver.scala:32)
[INFO] at scala.tools.nsc.Main$.doCompile(Main.scala:79)
[INFO] at scala.tools.nsc.Driver.process(Driver.scala:54)
[INFO] at scala.tools.nsc.Driver.main(Driver.scala:67)
[INFO] at scala.tools.nsc.Main.main(Main.scala)
[INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[INFO] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] at java.lang.reflect.Method.invoke(Method.java:606)
[INFO] at org_scala_tools_maven_executions.MainHelper.runMain(MainHelper.java:161)
[INFO] at org_scala_tools_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-216896889
Merging
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-214425488
Thanks for your contribution @stefanobortoli :-) I think the changes look good and can be merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-216475737
Modulo one more inline comment, I think it looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-216474116
Is this good to merge, @tillrohrmann?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-216793718
Thanks for the update @stefanobortoli.
PR is good to merge once tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-216525073
@stefanobortoli yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-215342703
@rmetzger said that it is related to corrupt cache files and that refreshing the caches would help. I think @uce already contacted INFRA to do that. Hopefully, this will fix the Scala problems.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1915
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Flink 1827 and small fixes in some tests
Posted by stefanobortoli <gi...@git.apache.org>.
Github user stefanobortoli commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-212305774
We had a look at the Flink test dependency hierarchies and the thing is feasible but I it should be done in a separate PR. In fact, it requires a clerical review of all projects and a better management of the test's transitive dependencies that right now is very redundant. For example, if I need _flink-optimizer_ I do not need to specify also the dependencies on _flink-core_ and _flink-runtime_ because they are included transitively.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Flink 1827 and small fixes in some tests
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-212076850
I quickly scrolled over the changed, and overall I like the idea of fixing the thing with the test utils.
I wonder if we could get rid of the "flink-test-utils" entirely by moving the contents into the "flink-tests" project. Have you considered that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1915#discussion_r60940770
--- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/taskmanager/TaskManagerStartupTest.java ---
@@ -53,41 +59,55 @@ public void testStartupWhenTaskmanagerActorPortIsUsed() {
blocker = new ServerSocket(0, 50, localAddress);
final int port = blocker.getLocalPort();
- try {
- TaskManager.runTaskManager(
- localHostName,
- ResourceID.generate(),
- port,
- new Configuration(),
+ TaskManager.runTaskManager(localHostName, ResourceID.generate(), port, new Configuration(),
TaskManager.class);
- fail("This should fail with an IOException");
- }
- catch (IOException e) {
- // expected. validate the error message
- assertNotNull(e.getMessage());
- assertTrue(e.getMessage().contains("Address already in use"));
+ fail("This should fail with an IOException");
+
+ } catch (IOException e) {
+ // expected. validate the error messagex
+ List<Throwable> causes = getExceptionCauses(e, new ArrayList<Throwable>());
+ for (Throwable cause : causes) {
+ if (cause instanceof BindException) {
+ throw (BindException) cause;
+ }
}
-
- }
- catch (Exception e) {
+ fail("This should fail with an exception caused by BindException");
+ } catch (Exception e) {
e.printStackTrace();
fail(e.getMessage());
- }
- finally {
+ } finally {
if (blocker != null) {
try {
blocker.close();
- }
- catch (IOException e) {
+ } catch (IOException e) {
// no need to log here
}
}
}
}
/**
- * Tests that the TaskManager startup fails synchronously when the I/O directories are
- * not writable.
+ * A utility method to analyze the exceptions and collect the clauses
+ *
+ * @param e
+ * the root exception (Throwable) object
+ * @param causes
+ * the list of exceptions that caused the root exceptions
+ * @return
+ */
+ private List<Throwable> getExceptionCauses(Throwable e, List<Throwable> causes) {
--- End diff --
What about factoring this method out into a Utils class so that it can be used by multiple tests. This would avoid code duplication.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: Flink 1827 and small fixes in some tests
Posted by stefanobortoli <gi...@git.apache.org>.
Github user stefanobortoli commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-212293101
We mainly wanted to simplify the build skipping tests. However, if you think it makes sense, we can give it a try.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1915#discussion_r61855640
--- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/util/StartupUtils.java ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.util;
+
+import java.util.List;
+
+public class StartupUtils {
+
+ /**
+ * A utility method to analyze the exceptions and collect the clauses
+ *
+ * @param e the root exception (Throwable) object
+ * @param causes the list of exceptions that caused the root exceptions
+ * @return a list of Throwable
+ */
+ public List<Throwable> getExceptionCauses(Throwable e, List<Throwable> causes) {
--- End diff --
I think it would be slightly better if we made this method `static`. Then we wouldn't have to create the `StartupUtils` objects in the different test classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by stefanobortoli <gi...@git.apache.org>.
Github user stefanobortoli commented on a diff in the pull request:
https://github.com/apache/flink/pull/1915#discussion_r61855824
--- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/util/StartupUtils.java ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.util;
+
+import java.util.List;
+
+public class StartupUtils {
+
+ /**
+ * A utility method to analyze the exceptions and collect the clauses
+ *
+ * @param e the root exception (Throwable) object
+ * @param causes the list of exceptions that caused the root exceptions
+ * @return a list of Throwable
+ */
+ public List<Throwable> getExceptionCauses(Throwable e, List<Throwable> causes) {
--- End diff --
on it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by thvasilo <gi...@git.apache.org>.
Github user thvasilo commented on the pull request:
https://github.com/apache/flink/pull/1915#issuecomment-215201773
Any idea what might be causing the error in FlinkML? I can't pinpoint it to any specific source file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1827] and small fixes in some tests
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/1915#discussion_r60942417
--- Diff: flink-test-utils/pom.xml ---
@@ -7,9 +7,7 @@ regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
-
--- End diff --
The license header looks to have been automatically reformatted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---