You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Dhruv Kumar <dh...@me.com> on 2016/04/16 19:15:14 UTC
Integration test failures
Hi Metron Team,
I’m Dhruv, I’d love to contribute to Metron’s development. I’m a Solutions Engineer at Hortonworks and I’m comfortable with Java, Storm, Maven, etc.
To kick things off I tried building and running Metron’s tests from metron-streaming on my mac but I got 2 test failures. I looked at test reports but couldn’t make out why this is happening. Is anyone else running into this issue? My system is: Oracle JDK 1.8.0_60, Maven 3.3.3, OS X 10.11.3
Failed tests: test(org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest)
test(org.apache.metron.dataloads.hbase.mr.LeastRecentlyUsedPrunerIntegrationTest)
Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Metron-Streaming ................................... SUCCESS [ 0.744 s]
[INFO] Metron-Common ...................................... SUCCESS [ 8.255 s]
[INFO] Metron-EnrichmentAdapters .......................... SUCCESS [ 0.763 s]
[INFO] Metron-MessageParsers .............................. SUCCESS [ 1.479 s]
[INFO] Metron-Indexing .................................... SUCCESS [ 0.540 s]
[INFO] Metron-Alerts ...................................... SUCCESS [ 0.792 s]
[INFO] Metron-Testing ..................................... SUCCESS [ 0.414 s]
[INFO] Metron-DataLoads ................................... FAILURE [04:50 min]
[INFO] Metron-Topologies .................................. SKIPPED
[INFO] Metron-Pcap_Service ................................ SKIPPED
For BulkLoadMapperIntegrationTest:
$ cat /Users/dkumar/code/incubator-metron-Metron_0.1BETA_rc7/metron-streaming/Metron-DataLoads/target/surefire-reports/org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest.txt
-------------------------------------------------------------------------------
Test set: org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.608 sec <<< FAILURE!
test(org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest) Time elapsed: 105.545 sec <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest.test(BulkLoadMapperIntegrationTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
Looks like some Hadoop job failed during the BulkLoadMapperIntegrationTest's run:
2016-04-16 09:40:00,953 WARN [ContainersLauncher #0] nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(213)) - Exception from container-launch with container ID: container_1460824746646_0001_02_000001 and exit code: 1
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
In the LeastRecentlyUsedPrunerIntegrationTest’s report, I see these type of messages indicating flaky Zookeeper connections:
2016-04-16 09:42:24,399 WARN [localhost:50688.activeMasterManager-SendThread(localhost:50719)] zookeeper.ClientCnxn (ClientCnxn.java:run(1102)) - Session 0x1541ff06a570005 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Any ideas? I can rerun and send more logs if needed.
Thanks,
Dhruv
Re: Integration test failures
Posted by James Sirota <js...@hortonworks.com>.
Hi Dhruv,
I was able to build just fine using the same setup as you. How are you pulling down the code? What maven commands are you running?
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Metron-Streaming ................................... SUCCESS [ 0.837 s]
[INFO] Metron-Common ...................................... SUCCESS [ 25.843 s]
[INFO] Metron-TestingUtilities ............................ SUCCESS [ 1.269 s]
[INFO] Metron-EnrichmentAdapters .......................... SUCCESS [ 1.750 s]
[INFO] Metron-MessageParsers .............................. SUCCESS [ 0.876 s]
[INFO] Metron-Indexing .................................... SUCCESS [ 0.193 s]
[INFO] Metron-Alerts ...................................... SUCCESS [ 0.722 s]
[INFO] Metron-Testing ..................................... SUCCESS [ 0.451 s]
[INFO] Metron-DataLoads ................................... SUCCESS [03:10 min]
[INFO] Metron-Topologies .................................. SUCCESS [ 34.402 s]
[INFO] Metron-Pcap_Service ................................ SUCCESS [ 16.449 s]
[INFO] Metron-Elasticsearch ............................... SUCCESS [ 37.009 s]
[INFO] Metron-Solr ........................................ SUCCESS [ 33.920 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:44 min
[INFO] Finished at: 2016-04-16T11:51:11-07:00
[INFO] Final Memory: 114M/1886M
[INFO] ------------------------------------------------------------------------
My environment is:
pache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T09:41:47-07:00)
Java version: 1.8.0_60, vendor: Oracle Corporation
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.4", arch: "x86_64", family: "mac"
Thanks,
James
On 4/16/16, 10:15 AM, "Dhruv Kumar" <dh...@me.com> wrote:
>Hi Metron Team,
>
>I’m Dhruv, I’d love to contribute to Metron’s development. I’m a Solutions Engineer at Hortonworks and I’m comfortable with Java, Storm, Maven, etc.
>
>To kick things off I tried building and running Metron’s tests from metron-streaming on my mac but I got 2 test failures. I looked at test reports but couldn’t make out why this is happening. Is anyone else running into this issue? My system is: Oracle JDK 1.8.0_60, Maven 3.3.3, OS X 10.11.3
>
>
>Failed tests: test(org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest)
> test(org.apache.metron.dataloads.hbase.mr.LeastRecentlyUsedPrunerIntegrationTest)
>
>Tests run: 3, Failures: 2, Errors: 0, Skipped: 0
>
>[INFO] ------------------------------------------------------------------------
>[INFO] Reactor Summary:
>[INFO]
>[INFO] Metron-Streaming ................................... SUCCESS [ 0.744 s]
>[INFO] Metron-Common ...................................... SUCCESS [ 8.255 s]
>[INFO] Metron-EnrichmentAdapters .......................... SUCCESS [ 0.763 s]
>[INFO] Metron-MessageParsers .............................. SUCCESS [ 1.479 s]
>[INFO] Metron-Indexing .................................... SUCCESS [ 0.540 s]
>[INFO] Metron-Alerts ...................................... SUCCESS [ 0.792 s]
>[INFO] Metron-Testing ..................................... SUCCESS [ 0.414 s]
>[INFO] Metron-DataLoads ................................... FAILURE [04:50 min]
>[INFO] Metron-Topologies .................................. SKIPPED
>[INFO] Metron-Pcap_Service ................................ SKIPPED
>
>
>
>For BulkLoadMapperIntegrationTest:
>
>$ cat /Users/dkumar/code/incubator-metron-Metron_0.1BETA_rc7/metron-streaming/Metron-DataLoads/target/surefire-reports/org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest.txt
>-------------------------------------------------------------------------------
>Test set: org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest
>-------------------------------------------------------------------------------
>Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 105.608 sec <<< FAILURE!
>test(org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest) Time elapsed: 105.545 sec <<< FAILURE!
>java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at org.apache.metron.dataloads.hbase.mr.BulkLoadMapperIntegrationTest.test(BulkLoadMapperIntegrationTest.java:95)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
>
>Looks like some Hadoop job failed during the BulkLoadMapperIntegrationTest's run:
>
>2016-04-16 09:40:00,953 WARN [ContainersLauncher #0] nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(213)) - Exception from container-launch with container ID: container_1460824746646_0001_02_000001 and exit code: 1
>ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>In the LeastRecentlyUsedPrunerIntegrationTest’s report, I see these type of messages indicating flaky Zookeeper connections:
>
>2016-04-16 09:42:24,399 WARN [localhost:50688.activeMasterManager-SendThread(localhost:50719)] zookeeper.ClientCnxn (ClientCnxn.java:run(1102)) - Session 0x1541ff06a570005 for server null, unexpected error, closing socket connection and attempting reconnect
>java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>
>Any ideas? I can rerun and send more logs if needed.
>
>Thanks,
>Dhruv
>
>
>