You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Christopher Thomas <ch...@gmail.com> on 2014/02/07 01:41:38 UTC
Problems building hadoop 2.2.0 from source
Hi,
I am a complete beginner to Hadoop, trying to build 2.2.0 from source on a
Macbook Pro running OS X Mavericks.
I am following the 'instructions' at
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
such
as they are.
I get the following test failure:
Forking command line: /bin/sh -c cd
/Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
&&
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
-Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
/Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
/Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
/Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
Results :
Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
[1:18:07.440s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 1:18:07.828s
[INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
[INFO] Final Memory: 32M/94M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
exception is java.util.concurrent.ExecutionException:
java.lang.RuntimeException: The forked VM terminated without saying
properly goodbye. VM crash or System.exit called ?
and the output from TestMRJobsWithHistoryService includes:
2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
delegation.AbstractDelegationTokenSecretManager
(AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
sleep interrupted
2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437)) -
Stopping server on 55131
2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
(Server.java:run(720)) - Stopping IPC Server listener on 55131
2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
(Server.java:run(866)) - Stopping IPC Server Responder
2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67)) -
Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
(JobHistory.java:serviceStop(136)) - Stopping JobHistory
2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
(JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67)) -
Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
2014-02-06 23:00:54,759 INFO [main] containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
Applications still running : [application_1391688017341_0001]
2014-02-06 23:00:54,762 INFO [main] containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
for Applications to be Finished
2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
application.Application (ApplicationImpl.java:handle(451)) - Application
application_1391688017341_0001 transitioned from RUNNING to
APPLICATION_RESOURCES_CLEANINGUP
2014-02-06 23:00:54,763 INFO [DeletionService #3]
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
:
/Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
2014-02-06 23:00:54,763 INFO [DeletionService #0]
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
:
/Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
APPLICATION_STOP for appId application_1391688017341_0001
2014-02-06 23:00:54,765 INFO [DeletionService #1]
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
:
/Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
application.Application (ApplicationImpl.java:handle(451)) - Application
application_1391688017341_0001 transitioned from
APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
loghandler.NonAggregatingLogHandler
(NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
application: application_1391688017341_0001, with delay of 10800 seconds
2014-02-06 23:00:54,768 INFO [DeletionService #3]
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
:
/Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
2014-02-06 23:00:58,766 INFO [main] containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
waiting for Applications to be Finished. Still alive:
[application_1391688017341_0001]
2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437)) -
Stopping server on 55125
2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
(Server.java:run(720)) - Stopping IPC Server listener on 55125
2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
(Server.java:run(866)) - Stopping IPC Server Responder
2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
unavailable on this system.
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
is disabled.
2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437)) -
Stopping server on 55126
2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
(Server.java:run(720)) - Stopping IPC Server listener on 55126
2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
(Server.java:run(866)) - Stopping IPC Server Responder
2014-02-06 23:01:08,785 INFO [Public Localizer]
localizer.ResourceLocalizationService
(ResourceLocalizationService.java:run(728)) - Public cache exiting
2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67)) -
Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
delegation.AbstractDelegationTokenSecretManager
(AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
sleep interrupted
2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
(MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
system...
2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
(MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
stopped.
2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
(MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
shutdown complete.
2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
amlauncher.ApplicationMasterLauncher
(ApplicationMasterLauncher.java:run(98)) -
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
interrupted. Returning.
2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437)) -
Stopping server on 55122
2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
(Server.java:run(720)) - Stopping IPC Server listener on 55122
2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437)) -
Stopping server on 55121
2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
(Server.java:run(866)) - Stopping IPC Server Responder
2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
(Server.java:run(720)) - Stopping IPC Server listener on 55121
2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
(Server.java:run(866)) - Stopping IPC Server Responder
2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437)) -
Stopping server on 55120
2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
(Server.java:run(720)) - Stopping IPC Server listener on 55120
2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
(Server.java:run(866)) - Stopping IPC Server Responder
2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437)) -
Stopping server on 55119
2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
(Server.java:run(720)) - Stopping IPC Server listener on 55119
2014-02-06 23:01:08,917 INFO [Ping Checker] util.AbstractLivelinessMonitor
(AbstractLivelinessMonitor.java:run(127)) - NMLivelinessMonitor thread
interrupted
2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
Returning, interrupted : java.lang.InterruptedException
2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
(Server.java:run(866)) - Stopping IPC Server Responder
2014-02-06 23:01:08,918 INFO [Ping Checker] util.AbstractLivelinessMonitor
(AbstractLivelinessMonitor.java:run(127)) - AMLivelinessMonitor thread
interrupted
2014-02-06 23:01:08,918 INFO [Ping Checker] util.AbstractLivelinessMonitor
(AbstractLivelinessMonitor.java:run(127)) -
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
thread interrupted
2014-02-06 23:01:08,918 INFO [Ping Checker] util.AbstractLivelinessMonitor
(AbstractLivelinessMonitor.java:run(127)) - AMLivelinessMonitor thread
interrupted
Has anyone seen this error before when building hadoop, and does anyone
have any ideas?
Regards
Chris
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
Thanks, I built 2.3 yesterday (checked out from from link suggested in
earlier post of this thread) without problems apart from VM running out of
memory which was fixed with
export MAVEN_OPTS=-Xmx2048m
At least, I got a message saying successful build.
Thanks for your help.
On 8 February 2014 10:53, Ted Yu <yu...@gmail.com> wrote:
> In the output for a passing test, I saw:
>
> 2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(557)) -
> InterruptedExcpetion recieved for ExpiredTokenRemover thread
> java.lang.InterruptedException: sleep interrupted
>
> Meaning the above was not critical.
>
> branch-2.3 is receiving attention now.
> Discovering test failure there would be more helpful.
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I guess the ERROR lines in
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService-output.txt
>>
>> led me to believe that the problem was with TestMRJobsWithHistoryService.
>> If that's not the case then what do these messages indicate?
>>
>> As I say I am a complete novice and the learning curve is very steep.
>>
>>
>> On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
>>
>>> I checked out source code from
>>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and
>>> it builds.
>>>
>>> From TestMRJobsWithHistoryService.txt, the test passed.
>>> What led to this test being singled out among the 454 tests ?
>>>
>>> Thanks
>>>
>>>
>>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Yes well I tried 2.3, but I have found a number of problems building
>>>> it. I had to resort to manually applying patches that I found in the bug
>>>> tracking lists, which did not seem to have made it into all branches. So
>>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>>> stable.
>>>>
>>>> I apologise for the confusion.
>>>>
>>>> Here is the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>>> though perhaps not that illuminating:
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>>> sec
>>>>
>>>>
>>>>
>>>>
>>>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> The output was
>>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>>
>>>>> Can you show us the contents
>>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> ?
>>>>>
>>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>>> more attention to hadoop 2.3
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> I included the last part of
>>>>>>
>>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>>> txt
>>>>>>
>>>>>> in the second half of my initial posting, calling it "the output
>>>>>> from TestMRJobsWithHistoryService". Sloppy terminology I know, sorry
>>>>>> if I wasn't very clear.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>>
>>>>>>> What
>>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>>> say ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from
>>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>>
>>>>>>>> I am following the 'instructions' at
>>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>>>> as they are.
>>>>>>>>
>>>>>>>> I get the following test failure:
>>>>>>>>
>>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>>> &&
>>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>>
>>>>>>>> Results :
>>>>>>>>
>>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>>
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Reactor Summary:
>>>>>>>> [INFO]
>>>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>>>> [1:18:07.440s]
>>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>>
>>>>>>>>
>>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>>>> for Applications to be Finished
>>>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>>> [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>>>> unavailable on this system.
>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>>> is disabled.
>>>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>>>> localizer.ResourceLocalizationService
>>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>>>> system...
>>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>>>> stopped.
>>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>>>> shutdown complete.
>>>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>>> interrupted. Returning.
>>>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>>> thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>>
>>>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>>>> anyone have any ideas?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
Thanks, I built 2.3 yesterday (checked out from from link suggested in
earlier post of this thread) without problems apart from VM running out of
memory which was fixed with
export MAVEN_OPTS=-Xmx2048m
At least, I got a message saying successful build.
Thanks for your help.
On 8 February 2014 10:53, Ted Yu <yu...@gmail.com> wrote:
> In the output for a passing test, I saw:
>
> 2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(557)) -
> InterruptedExcpetion recieved for ExpiredTokenRemover thread
> java.lang.InterruptedException: sleep interrupted
>
> Meaning the above was not critical.
>
> branch-2.3 is receiving attention now.
> Discovering test failure there would be more helpful.
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I guess the ERROR lines in
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService-output.txt
>>
>> led me to believe that the problem was with TestMRJobsWithHistoryService.
>> If that's not the case then what do these messages indicate?
>>
>> As I say I am a complete novice and the learning curve is very steep.
>>
>>
>> On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
>>
>>> I checked out source code from
>>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and
>>> it builds.
>>>
>>> From TestMRJobsWithHistoryService.txt, the test passed.
>>> What led to this test being singled out among the 454 tests ?
>>>
>>> Thanks
>>>
>>>
>>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Yes well I tried 2.3, but I have found a number of problems building
>>>> it. I had to resort to manually applying patches that I found in the bug
>>>> tracking lists, which did not seem to have made it into all branches. So
>>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>>> stable.
>>>>
>>>> I apologise for the confusion.
>>>>
>>>> Here is the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>>> though perhaps not that illuminating:
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>>> sec
>>>>
>>>>
>>>>
>>>>
>>>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> The output was
>>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>>
>>>>> Can you show us the contents
>>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> ?
>>>>>
>>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>>> more attention to hadoop 2.3
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> I included the last part of
>>>>>>
>>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>>> txt
>>>>>>
>>>>>> in the second half of my initial posting, calling it "the output
>>>>>> from TestMRJobsWithHistoryService". Sloppy terminology I know, sorry
>>>>>> if I wasn't very clear.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>>
>>>>>>> What
>>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>>> say ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from
>>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>>
>>>>>>>> I am following the 'instructions' at
>>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>>>> as they are.
>>>>>>>>
>>>>>>>> I get the following test failure:
>>>>>>>>
>>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>>> &&
>>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>>
>>>>>>>> Results :
>>>>>>>>
>>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>>
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Reactor Summary:
>>>>>>>> [INFO]
>>>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>>>> [1:18:07.440s]
>>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>>
>>>>>>>>
>>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>>>> for Applications to be Finished
>>>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>>> [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>>>> unavailable on this system.
>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>>> is disabled.
>>>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>>>> localizer.ResourceLocalizationService
>>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>>>> system...
>>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>>>> stopped.
>>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>>>> shutdown complete.
>>>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>>> interrupted. Returning.
>>>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>>> thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>>
>>>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>>>> anyone have any ideas?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
Thanks, I built 2.3 yesterday (checked out from from link suggested in
earlier post of this thread) without problems apart from VM running out of
memory which was fixed with
export MAVEN_OPTS=-Xmx2048m
At least, I got a message saying successful build.
Thanks for your help.
On 8 February 2014 10:53, Ted Yu <yu...@gmail.com> wrote:
> In the output for a passing test, I saw:
>
> 2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(557)) -
> InterruptedExcpetion recieved for ExpiredTokenRemover thread
> java.lang.InterruptedException: sleep interrupted
>
> Meaning the above was not critical.
>
> branch-2.3 is receiving attention now.
> Discovering test failure there would be more helpful.
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I guess the ERROR lines in
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService-output.txt
>>
>> led me to believe that the problem was with TestMRJobsWithHistoryService.
>> If that's not the case then what do these messages indicate?
>>
>> As I say I am a complete novice and the learning curve is very steep.
>>
>>
>> On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
>>
>>> I checked out source code from
>>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and
>>> it builds.
>>>
>>> From TestMRJobsWithHistoryService.txt, the test passed.
>>> What led to this test being singled out among the 454 tests ?
>>>
>>> Thanks
>>>
>>>
>>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Yes well I tried 2.3, but I have found a number of problems building
>>>> it. I had to resort to manually applying patches that I found in the bug
>>>> tracking lists, which did not seem to have made it into all branches. So
>>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>>> stable.
>>>>
>>>> I apologise for the confusion.
>>>>
>>>> Here is the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>>> though perhaps not that illuminating:
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>>> sec
>>>>
>>>>
>>>>
>>>>
>>>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> The output was
>>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>>
>>>>> Can you show us the contents
>>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> ?
>>>>>
>>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>>> more attention to hadoop 2.3
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> I included the last part of
>>>>>>
>>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>>> txt
>>>>>>
>>>>>> in the second half of my initial posting, calling it "the output
>>>>>> from TestMRJobsWithHistoryService". Sloppy terminology I know, sorry
>>>>>> if I wasn't very clear.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>>
>>>>>>> What
>>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>>> say ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from
>>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>>
>>>>>>>> I am following the 'instructions' at
>>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>>>> as they are.
>>>>>>>>
>>>>>>>> I get the following test failure:
>>>>>>>>
>>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>>> &&
>>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>>
>>>>>>>> Results :
>>>>>>>>
>>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>>
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Reactor Summary:
>>>>>>>> [INFO]
>>>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>>>> [1:18:07.440s]
>>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>>
>>>>>>>>
>>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>>>> for Applications to be Finished
>>>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>>> [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>>>> unavailable on this system.
>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>>> is disabled.
>>>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>>>> localizer.ResourceLocalizationService
>>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>>>> system...
>>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>>>> stopped.
>>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>>>> shutdown complete.
>>>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>>> interrupted. Returning.
>>>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>>> thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>>
>>>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>>>> anyone have any ideas?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
Thanks, I built 2.3 yesterday (checked out from from link suggested in
earlier post of this thread) without problems apart from VM running out of
memory which was fixed with
export MAVEN_OPTS=-Xmx2048m
At least, I got a message saying successful build.
Thanks for your help.
On 8 February 2014 10:53, Ted Yu <yu...@gmail.com> wrote:
> In the output for a passing test, I saw:
>
> 2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(557)) -
> InterruptedExcpetion recieved for ExpiredTokenRemover thread
> java.lang.InterruptedException: sleep interrupted
>
> Meaning the above was not critical.
>
> branch-2.3 is receiving attention now.
> Discovering test failure there would be more helpful.
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I guess the ERROR lines in
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService-output.txt
>>
>> led me to believe that the problem was with TestMRJobsWithHistoryService.
>> If that's not the case then what do these messages indicate?
>>
>> As I say I am a complete novice and the learning curve is very steep.
>>
>>
>> On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
>>
>>> I checked out source code from
>>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and
>>> it builds.
>>>
>>> From TestMRJobsWithHistoryService.txt, the test passed.
>>> What led to this test being singled out among the 454 tests ?
>>>
>>> Thanks
>>>
>>>
>>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Yes well I tried 2.3, but I have found a number of problems building
>>>> it. I had to resort to manually applying patches that I found in the bug
>>>> tracking lists, which did not seem to have made it into all branches. So
>>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>>> stable.
>>>>
>>>> I apologise for the confusion.
>>>>
>>>> Here is the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>>> though perhaps not that illuminating:
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>>> sec
>>>>
>>>>
>>>>
>>>>
>>>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> The output was
>>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>>
>>>>> Can you show us the contents
>>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> ?
>>>>>
>>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>>> more attention to hadoop 2.3
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> I included the last part of
>>>>>>
>>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>>> txt
>>>>>>
>>>>>> in the second half of my initial posting, calling it "the output
>>>>>> from TestMRJobsWithHistoryService". Sloppy terminology I know, sorry
>>>>>> if I wasn't very clear.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>>
>>>>>>> What
>>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>>> say ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from
>>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>>
>>>>>>>> I am following the 'instructions' at
>>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>>>> as they are.
>>>>>>>>
>>>>>>>> I get the following test failure:
>>>>>>>>
>>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>>> &&
>>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>>
>>>>>>>> Results :
>>>>>>>>
>>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>>
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Reactor Summary:
>>>>>>>> [INFO]
>>>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>>>> [1:18:07.440s]
>>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>>
>>>>>>>>
>>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>>>> for Applications to be Finished
>>>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>>> [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>>>> unavailable on this system.
>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>>> is disabled.
>>>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>>>> localizer.ResourceLocalizationService
>>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>>>> system...
>>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>>>> stopped.
>>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>>>> shutdown complete.
>>>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>>> interrupted. Returning.
>>>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>>> thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>>
>>>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>>>> anyone have any ideas?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
In the output for a passing test, I saw:
2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
delegation.AbstractDelegationTokenSecretManager
(AbstractDelegationTokenSecretManager.java:run(557)) -
InterruptedExcpetion recieved for ExpiredTokenRemover thread
java.lang.InterruptedException: sleep interrupted
Meaning the above was not critical.
branch-2.3 is receiving attention now.
Discovering test failure there would be more helpful.
Cheers
On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> I guess the ERROR lines in
>
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
> TestMRJobsWithHistoryService-output.txt
>
> led me to believe that the problem was with TestMRJobsWithHistoryService.
> If that's not the case then what do these messages indicate?
>
> As I say I am a complete novice and the learning curve is very steep.
>
>
> On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
>
>> I checked out source code from
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
>> builds.
>>
>> From TestMRJobsWithHistoryService.txt, the test passed.
>> What led to this test being singled out among the 454 tests ?
>>
>> Thanks
>>
>>
>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> Yes well I tried 2.3, but I have found a number of problems building it.
>>> I had to resort to manually applying patches that I found in the bug
>>> tracking lists, which did not seem to have made it into all branches. So
>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>> stable.
>>>
>>> I apologise for the confusion.
>>>
>>> Here is the contents
>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>> though perhaps not that illuminating:
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>> sec
>>>
>>>
>>>
>>>
>>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The output was
>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>
>>>> Can you show us the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> ?
>>>>
>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>> more attention to hadoop 2.3
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> I included the last part of
>>>>>
>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>> txt
>>>>>
>>>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>
>>>>>> What
>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>> say ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from
>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>
>>>>>>> I am following the 'instructions' at
>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>>> as they are.
>>>>>>>
>>>>>>> I get the following test failure:
>>>>>>>
>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>> &&
>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>
>>>>>>> Results :
>>>>>>>
>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Reactor Summary:
>>>>>>> [INFO]
>>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>>> [1:18:07.440s]
>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] BUILD FAILURE
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [ERROR] Failed to execute goal
>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>
>>>>>>>
>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>
>>>>>>>
>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>> sleep interrupted
>>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>>> for Applications to be Finished
>>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>> [application_1391688017341_0001]
>>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>>> unavailable on this system.
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>> is disabled.
>>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>>> localizer.ResourceLocalizationService
>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>> sleep interrupted
>>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>>> system...
>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>>> stopped.
>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>>> shutdown complete.
>>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>> interrupted. Returning.
>>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>> thread interrupted
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>
>>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>>> anyone have any ideas?
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
In the output for a passing test, I saw:
2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
delegation.AbstractDelegationTokenSecretManager
(AbstractDelegationTokenSecretManager.java:run(557)) -
InterruptedExcpetion recieved for ExpiredTokenRemover thread
java.lang.InterruptedException: sleep interrupted
Meaning the above was not critical.
branch-2.3 is receiving attention now.
Discovering test failure there would be more helpful.
Cheers
On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> I guess the ERROR lines in
>
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
> TestMRJobsWithHistoryService-output.txt
>
> led me to believe that the problem was with TestMRJobsWithHistoryService.
> If that's not the case then what do these messages indicate?
>
> As I say I am a complete novice and the learning curve is very steep.
>
>
> On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
>
>> I checked out source code from
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
>> builds.
>>
>> From TestMRJobsWithHistoryService.txt, the test passed.
>> What led to this test being singled out among the 454 tests ?
>>
>> Thanks
>>
>>
>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> Yes well I tried 2.3, but I have found a number of problems building it.
>>> I had to resort to manually applying patches that I found in the bug
>>> tracking lists, which did not seem to have made it into all branches. So
>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>> stable.
>>>
>>> I apologise for the confusion.
>>>
>>> Here is the contents
>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>> though perhaps not that illuminating:
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>> sec
>>>
>>>
>>>
>>>
>>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The output was
>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>
>>>> Can you show us the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> ?
>>>>
>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>> more attention to hadoop 2.3
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> I included the last part of
>>>>>
>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>> txt
>>>>>
>>>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>
>>>>>> What
>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>> say ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from
>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>
>>>>>>> I am following the 'instructions' at
>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>>> as they are.
>>>>>>>
>>>>>>> I get the following test failure:
>>>>>>>
>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>> &&
>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>
>>>>>>> Results :
>>>>>>>
>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Reactor Summary:
>>>>>>> [INFO]
>>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>>> [1:18:07.440s]
>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] BUILD FAILURE
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [ERROR] Failed to execute goal
>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>
>>>>>>>
>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>
>>>>>>>
>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>> sleep interrupted
>>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>>> for Applications to be Finished
>>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>> [application_1391688017341_0001]
>>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>>> unavailable on this system.
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>> is disabled.
>>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>>> localizer.ResourceLocalizationService
>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>> sleep interrupted
>>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>>> system...
>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>>> stopped.
>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>>> shutdown complete.
>>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>> interrupted. Returning.
>>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>> thread interrupted
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>
>>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>>> anyone have any ideas?
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
In the output for a passing test, I saw:
2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
delegation.AbstractDelegationTokenSecretManager
(AbstractDelegationTokenSecretManager.java:run(557)) -
InterruptedExcpetion recieved for ExpiredTokenRemover thread
java.lang.InterruptedException: sleep interrupted
Meaning the above was not critical.
branch-2.3 is receiving attention now.
Discovering test failure there would be more helpful.
Cheers
On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> I guess the ERROR lines in
>
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
> TestMRJobsWithHistoryService-output.txt
>
> led me to believe that the problem was with TestMRJobsWithHistoryService.
> If that's not the case then what do these messages indicate?
>
> As I say I am a complete novice and the learning curve is very steep.
>
>
> On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
>
>> I checked out source code from
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
>> builds.
>>
>> From TestMRJobsWithHistoryService.txt, the test passed.
>> What led to this test being singled out among the 454 tests ?
>>
>> Thanks
>>
>>
>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> Yes well I tried 2.3, but I have found a number of problems building it.
>>> I had to resort to manually applying patches that I found in the bug
>>> tracking lists, which did not seem to have made it into all branches. So
>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>> stable.
>>>
>>> I apologise for the confusion.
>>>
>>> Here is the contents
>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>> though perhaps not that illuminating:
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>> sec
>>>
>>>
>>>
>>>
>>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The output was
>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>
>>>> Can you show us the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> ?
>>>>
>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>> more attention to hadoop 2.3
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> I included the last part of
>>>>>
>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>> txt
>>>>>
>>>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>
>>>>>> What
>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>> say ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from
>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>
>>>>>>> I am following the 'instructions' at
>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>>> as they are.
>>>>>>>
>>>>>>> I get the following test failure:
>>>>>>>
>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>> &&
>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>
>>>>>>> Results :
>>>>>>>
>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Reactor Summary:
>>>>>>> [INFO]
>>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>>> [1:18:07.440s]
>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] BUILD FAILURE
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [ERROR] Failed to execute goal
>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>
>>>>>>>
>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>
>>>>>>>
>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>> sleep interrupted
>>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>>> for Applications to be Finished
>>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>> [application_1391688017341_0001]
>>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>>> unavailable on this system.
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>> is disabled.
>>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>>> localizer.ResourceLocalizationService
>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>> sleep interrupted
>>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>>> system...
>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>>> stopped.
>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>>> shutdown complete.
>>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>> interrupted. Returning.
>>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>> thread interrupted
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>
>>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>>> anyone have any ideas?
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
In the output for a passing test, I saw:
2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
delegation.AbstractDelegationTokenSecretManager
(AbstractDelegationTokenSecretManager.java:run(557)) -
InterruptedExcpetion recieved for ExpiredTokenRemover thread
java.lang.InterruptedException: sleep interrupted
Meaning the above was not critical.
branch-2.3 is receiving attention now.
Discovering test failure there would be more helpful.
Cheers
On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> I guess the ERROR lines in
>
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
> TestMRJobsWithHistoryService-output.txt
>
> led me to believe that the problem was with TestMRJobsWithHistoryService.
> If that's not the case then what do these messages indicate?
>
> As I say I am a complete novice and the learning curve is very steep.
>
>
> On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
>
>> I checked out source code from
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
>> builds.
>>
>> From TestMRJobsWithHistoryService.txt, the test passed.
>> What led to this test being singled out among the 454 tests ?
>>
>> Thanks
>>
>>
>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> Yes well I tried 2.3, but I have found a number of problems building it.
>>> I had to resort to manually applying patches that I found in the bug
>>> tracking lists, which did not seem to have made it into all branches. So
>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>> stable.
>>>
>>> I apologise for the confusion.
>>>
>>> Here is the contents
>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>> though perhaps not that illuminating:
>>>
>>>
>>>
>>> -------------------------------------------------------------------------------
>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>
>>> -------------------------------------------------------------------------------
>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>> sec
>>>
>>>
>>>
>>>
>>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The output was
>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>
>>>> Can you show us the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> ?
>>>>
>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>> more attention to hadoop 2.3
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> I included the last part of
>>>>>
>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>> txt
>>>>>
>>>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>>
>>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>
>>>>>> What
>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>> say ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from
>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>
>>>>>>> I am following the 'instructions' at
>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>>> as they are.
>>>>>>>
>>>>>>> I get the following test failure:
>>>>>>>
>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>> &&
>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>
>>>>>>> Results :
>>>>>>>
>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Reactor Summary:
>>>>>>> [INFO]
>>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>>> [1:18:07.440s]
>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] BUILD FAILURE
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>> [INFO]
>>>>>>> ------------------------------------------------------------------------
>>>>>>> [ERROR] Failed to execute goal
>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>
>>>>>>>
>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>
>>>>>>>
>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>> sleep interrupted
>>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>>> for Applications to be Finished
>>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>>> :
>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>>> containermanager.ContainerManagerImpl
>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>> [application_1391688017341_0001]
>>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>>> unavailable on this system.
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>> is disabled.
>>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>>> localizer.ResourceLocalizationService
>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>> sleep interrupted
>>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>>> system...
>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>>> stopped.
>>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>>> shutdown complete.
>>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>> interrupted. Returning.
>>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>> thread interrupted
>>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>
>>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>>> anyone have any ideas?
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
I guess the ERROR lines in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
TestMRJobsWithHistoryService-output.txt
led me to believe that the problem was with TestMRJobsWithHistoryService.
If that's not the case then what do these messages indicate?
As I say I am a complete novice and the learning curve is very steep.
On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
> I checked out source code from
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
> builds.
>
> From TestMRJobsWithHistoryService.txt, the test passed.
> What led to this test being singled out among the 454 tests ?
>
> Thanks
>
>
> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> Yes well I tried 2.3, but I have found a number of problems building it.
>> I had to resort to manually applying patches that I found in the bug
>> tracking lists, which did not seem to have made it into all branches. So
>> for the moment I am sticking with 2.2.0 which is advertised as being
>> stable.
>>
>> I apologise for the confusion.
>>
>> Here is the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>> though perhaps not that illuminating:
>>
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>
>> -------------------------------------------------------------------------------
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>>
>>
>>
>>
>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>
>>> The output was
>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>
>>> Can you show us the contents
>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>> ?
>>>
>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>> more attention to hadoop 2.3
>>>
>>> Cheers
>>>
>>>
>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> I included the last part of
>>>>
>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>> txt
>>>>
>>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>>
>>>> Regards
>>>>
>>>> Chris
>>>>
>>>>
>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>
>>>>> What
>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> say ?
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>>
>>>>>> I am following the 'instructions' at
>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>> as they are.
>>>>>>
>>>>>> I get the following test failure:
>>>>>>
>>>>>> Forking command line: /bin/sh -c cd
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>> &&
>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>
>>>>>> Results :
>>>>>>
>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Reactor Summary:
>>>>>> [INFO]
>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>> [1:18:07.440s]
>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] BUILD FAILURE
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>> [INFO] Final Memory: 32M/94M
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [ERROR] Failed to execute goal
>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>
>>>>>>
>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>
>>>>>>
>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>> sleep interrupted
>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>> for Applications to be Finished
>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>> application_1391688017341_0001 transitioned from
>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>> loghandler.NonAggregatingLogHandler
>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>> [application_1391688017341_0001]
>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>> unavailable on this system.
>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>> is disabled.
>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>> localizer.ResourceLocalizationService
>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>> sleep interrupted
>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>> system...
>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>> stopped.
>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>> shutdown complete.
>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>> interrupted. Returning.
>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> NMLivelinessMonitor thread interrupted
>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> AMLivelinessMonitor thread interrupted
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>> thread interrupted
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>
>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>> anyone have any ideas?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
I guess the ERROR lines in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
TestMRJobsWithHistoryService-output.txt
led me to believe that the problem was with TestMRJobsWithHistoryService.
If that's not the case then what do these messages indicate?
As I say I am a complete novice and the learning curve is very steep.
On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
> I checked out source code from
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
> builds.
>
> From TestMRJobsWithHistoryService.txt, the test passed.
> What led to this test being singled out among the 454 tests ?
>
> Thanks
>
>
> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> Yes well I tried 2.3, but I have found a number of problems building it.
>> I had to resort to manually applying patches that I found in the bug
>> tracking lists, which did not seem to have made it into all branches. So
>> for the moment I am sticking with 2.2.0 which is advertised as being
>> stable.
>>
>> I apologise for the confusion.
>>
>> Here is the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>> though perhaps not that illuminating:
>>
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>
>> -------------------------------------------------------------------------------
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>>
>>
>>
>>
>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>
>>> The output was
>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>
>>> Can you show us the contents
>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>> ?
>>>
>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>> more attention to hadoop 2.3
>>>
>>> Cheers
>>>
>>>
>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> I included the last part of
>>>>
>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>> txt
>>>>
>>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>>
>>>> Regards
>>>>
>>>> Chris
>>>>
>>>>
>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>
>>>>> What
>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> say ?
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>>
>>>>>> I am following the 'instructions' at
>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>> as they are.
>>>>>>
>>>>>> I get the following test failure:
>>>>>>
>>>>>> Forking command line: /bin/sh -c cd
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>> &&
>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>
>>>>>> Results :
>>>>>>
>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Reactor Summary:
>>>>>> [INFO]
>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>> [1:18:07.440s]
>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] BUILD FAILURE
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>> [INFO] Final Memory: 32M/94M
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [ERROR] Failed to execute goal
>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>
>>>>>>
>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>
>>>>>>
>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>> sleep interrupted
>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>> for Applications to be Finished
>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>> application_1391688017341_0001 transitioned from
>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>> loghandler.NonAggregatingLogHandler
>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>> [application_1391688017341_0001]
>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>> unavailable on this system.
>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>> is disabled.
>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>> localizer.ResourceLocalizationService
>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>> sleep interrupted
>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>> system...
>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>> stopped.
>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>> shutdown complete.
>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>> interrupted. Returning.
>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> NMLivelinessMonitor thread interrupted
>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> AMLivelinessMonitor thread interrupted
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>> thread interrupted
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>
>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>> anyone have any ideas?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
I guess the ERROR lines in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
TestMRJobsWithHistoryService-output.txt
led me to believe that the problem was with TestMRJobsWithHistoryService.
If that's not the case then what do these messages indicate?
As I say I am a complete novice and the learning curve is very steep.
On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
> I checked out source code from
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
> builds.
>
> From TestMRJobsWithHistoryService.txt, the test passed.
> What led to this test being singled out among the 454 tests ?
>
> Thanks
>
>
> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> Yes well I tried 2.3, but I have found a number of problems building it.
>> I had to resort to manually applying patches that I found in the bug
>> tracking lists, which did not seem to have made it into all branches. So
>> for the moment I am sticking with 2.2.0 which is advertised as being
>> stable.
>>
>> I apologise for the confusion.
>>
>> Here is the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>> though perhaps not that illuminating:
>>
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>
>> -------------------------------------------------------------------------------
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>>
>>
>>
>>
>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>
>>> The output was
>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>
>>> Can you show us the contents
>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>> ?
>>>
>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>> more attention to hadoop 2.3
>>>
>>> Cheers
>>>
>>>
>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> I included the last part of
>>>>
>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>> txt
>>>>
>>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>>
>>>> Regards
>>>>
>>>> Chris
>>>>
>>>>
>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>
>>>>> What
>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> say ?
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>>
>>>>>> I am following the 'instructions' at
>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>> as they are.
>>>>>>
>>>>>> I get the following test failure:
>>>>>>
>>>>>> Forking command line: /bin/sh -c cd
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>> &&
>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>
>>>>>> Results :
>>>>>>
>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Reactor Summary:
>>>>>> [INFO]
>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>> [1:18:07.440s]
>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] BUILD FAILURE
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>> [INFO] Final Memory: 32M/94M
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [ERROR] Failed to execute goal
>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>
>>>>>>
>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>
>>>>>>
>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>> sleep interrupted
>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>> for Applications to be Finished
>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>> application_1391688017341_0001 transitioned from
>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>> loghandler.NonAggregatingLogHandler
>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>> [application_1391688017341_0001]
>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>> unavailable on this system.
>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>> is disabled.
>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>> localizer.ResourceLocalizationService
>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>> sleep interrupted
>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>> system...
>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>> stopped.
>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>> shutdown complete.
>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>> interrupted. Returning.
>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> NMLivelinessMonitor thread interrupted
>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> AMLivelinessMonitor thread interrupted
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>> thread interrupted
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>
>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>> anyone have any ideas?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
I guess the ERROR lines in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
TestMRJobsWithHistoryService-output.txt
led me to believe that the problem was with TestMRJobsWithHistoryService.
If that's not the case then what do these messages indicate?
As I say I am a complete novice and the learning curve is very steep.
On 7 February 2014 14:47, Ted Yu <yu...@gmail.com> wrote:
> I checked out source code from
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
> builds.
>
> From TestMRJobsWithHistoryService.txt, the test passed.
> What led to this test being singled out among the 454 tests ?
>
> Thanks
>
>
> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> Yes well I tried 2.3, but I have found a number of problems building it.
>> I had to resort to manually applying patches that I found in the bug
>> tracking lists, which did not seem to have made it into all branches. So
>> for the moment I am sticking with 2.2.0 which is advertised as being
>> stable.
>>
>> I apologise for the confusion.
>>
>> Here is the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>> though perhaps not that illuminating:
>>
>>
>>
>> -------------------------------------------------------------------------------
>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>
>> -------------------------------------------------------------------------------
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>>
>>
>>
>>
>> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>>
>>> The output was
>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>
>>> Can you show us the contents
>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>> ?
>>>
>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>> more attention to hadoop 2.3
>>>
>>> Cheers
>>>
>>>
>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> I included the last part of
>>>>
>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>> txt
>>>>
>>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>>
>>>> Regards
>>>>
>>>> Chris
>>>>
>>>>
>>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>
>>>>> What
>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> say ?
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>>
>>>>>> I am following the 'instructions' at
>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>>> as they are.
>>>>>>
>>>>>> I get the following test failure:
>>>>>>
>>>>>> Forking command line: /bin/sh -c cd
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>> &&
>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>
>>>>>> Results :
>>>>>>
>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Reactor Summary:
>>>>>> [INFO]
>>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>>> [1:18:07.440s]
>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] BUILD FAILURE
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>> [INFO] Final Memory: 32M/94M
>>>>>> [INFO]
>>>>>> ------------------------------------------------------------------------
>>>>>> [ERROR] Failed to execute goal
>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>
>>>>>>
>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>
>>>>>>
>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>> sleep interrupted
>>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>>> for Applications to be Finished
>>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>>> application_1391688017341_0001 transitioned from
>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>>> loghandler.NonAggregatingLogHandler
>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>>> nodemanager.DefaultContainerExecutor
>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>>> :
>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>>> containermanager.ContainerManagerImpl
>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>> [application_1391688017341_0001]
>>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>>> unavailable on this system.
>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>> is disabled.
>>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>>> localizer.ResourceLocalizationService
>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>> sleep interrupted
>>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>>> system...
>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>>> stopped.
>>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>>> shutdown complete.
>>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>> interrupted. Returning.
>>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> NMLivelinessMonitor thread interrupted
>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> AMLivelinessMonitor thread interrupted
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>> thread interrupted
>>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>
>>>>>> Has anyone seen this error before when building hadoop, and does
>>>>>> anyone have any ideas?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
I checked out source code from
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
builds.
>From TestMRJobsWithHistoryService.txt, the test passed.
What led to this test being singled out among the 454 tests ?
Thanks
On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> Yes well I tried 2.3, but I have found a number of problems building it. I
> had to resort to manually applying patches that I found in the bug tracking
> lists, which did not seem to have made it into all branches. So for the
> moment I am sticking with 2.2.0 which is advertised as being stable.
>
> I apologise for the confusion.
>
> Here is the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
> though perhaps not that illuminating:
>
>
>
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>
>
>
>
> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>
>> The output was
>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>
>> Can you show us the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> ?
>>
>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>> more attention to hadoop 2.3
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> I included the last part of
>>>
>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>
>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>
>>> Regards
>>>
>>> Chris
>>>
>>>
>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>
>>>> What
>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> say ?
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>
>>>>> I am following the 'instructions' at
>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>> as they are.
>>>>>
>>>>> I get the following test failure:
>>>>>
>>>>> Forking command line: /bin/sh -c cd
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>> &&
>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>
>>>>> Results :
>>>>>
>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Reactor Summary:
>>>>> [INFO]
>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>> [1:18:07.440s]
>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 1:18:07.828s
>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>> [INFO] Final Memory: 32M/94M
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [ERROR] Failed to execute goal
>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>> exception is java.util.concurrent.ExecutionException:
>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>
>>>>>
>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>
>>>>>
>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>> Applications still running : [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>> for Applications to be Finished
>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from
>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>> loghandler.NonAggregatingLogHandler
>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>> waiting for Applications to be Finished. Still alive:
>>>>> [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>> unavailable on this system.
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>> is disabled.
>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>> localizer.ResourceLocalizationService
>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>> system...
>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>> stopped.
>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>> shutdown complete.
>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>> amlauncher.ApplicationMasterLauncher
>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>> interrupted. Returning.
>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> NMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> AMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>> thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> AMLivelinessMonitor thread interrupted
>>>>>
>>>>> Has anyone seen this error before when building hadoop, and does
>>>>> anyone have any ideas?
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
I checked out source code from
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
builds.
>From TestMRJobsWithHistoryService.txt, the test passed.
What led to this test being singled out among the 454 tests ?
Thanks
On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> Yes well I tried 2.3, but I have found a number of problems building it. I
> had to resort to manually applying patches that I found in the bug tracking
> lists, which did not seem to have made it into all branches. So for the
> moment I am sticking with 2.2.0 which is advertised as being stable.
>
> I apologise for the confusion.
>
> Here is the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
> though perhaps not that illuminating:
>
>
>
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>
>
>
>
> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>
>> The output was
>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>
>> Can you show us the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> ?
>>
>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>> more attention to hadoop 2.3
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> I included the last part of
>>>
>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>
>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>
>>> Regards
>>>
>>> Chris
>>>
>>>
>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>
>>>> What
>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> say ?
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>
>>>>> I am following the 'instructions' at
>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>> as they are.
>>>>>
>>>>> I get the following test failure:
>>>>>
>>>>> Forking command line: /bin/sh -c cd
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>> &&
>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>
>>>>> Results :
>>>>>
>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Reactor Summary:
>>>>> [INFO]
>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>> [1:18:07.440s]
>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 1:18:07.828s
>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>> [INFO] Final Memory: 32M/94M
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [ERROR] Failed to execute goal
>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>> exception is java.util.concurrent.ExecutionException:
>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>
>>>>>
>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>
>>>>>
>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>> Applications still running : [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>> for Applications to be Finished
>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from
>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>> loghandler.NonAggregatingLogHandler
>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>> waiting for Applications to be Finished. Still alive:
>>>>> [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>> unavailable on this system.
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>> is disabled.
>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>> localizer.ResourceLocalizationService
>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>> system...
>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>> stopped.
>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>> shutdown complete.
>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>> amlauncher.ApplicationMasterLauncher
>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>> interrupted. Returning.
>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> NMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> AMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>> thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> AMLivelinessMonitor thread interrupted
>>>>>
>>>>> Has anyone seen this error before when building hadoop, and does
>>>>> anyone have any ideas?
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
I checked out source code from
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
builds.
>From TestMRJobsWithHistoryService.txt, the test passed.
What led to this test being singled out among the 454 tests ?
Thanks
On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> Yes well I tried 2.3, but I have found a number of problems building it. I
> had to resort to manually applying patches that I found in the bug tracking
> lists, which did not seem to have made it into all branches. So for the
> moment I am sticking with 2.2.0 which is advertised as being stable.
>
> I apologise for the confusion.
>
> Here is the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
> though perhaps not that illuminating:
>
>
>
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>
>
>
>
> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>
>> The output was
>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>
>> Can you show us the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> ?
>>
>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>> more attention to hadoop 2.3
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> I included the last part of
>>>
>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>
>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>
>>> Regards
>>>
>>> Chris
>>>
>>>
>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>
>>>> What
>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> say ?
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>
>>>>> I am following the 'instructions' at
>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>> as they are.
>>>>>
>>>>> I get the following test failure:
>>>>>
>>>>> Forking command line: /bin/sh -c cd
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>> &&
>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>
>>>>> Results :
>>>>>
>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Reactor Summary:
>>>>> [INFO]
>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>> [1:18:07.440s]
>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 1:18:07.828s
>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>> [INFO] Final Memory: 32M/94M
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [ERROR] Failed to execute goal
>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>> exception is java.util.concurrent.ExecutionException:
>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>
>>>>>
>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>
>>>>>
>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>> Applications still running : [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>> for Applications to be Finished
>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from
>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>> loghandler.NonAggregatingLogHandler
>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>> waiting for Applications to be Finished. Still alive:
>>>>> [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>> unavailable on this system.
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>> is disabled.
>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>> localizer.ResourceLocalizationService
>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>> system...
>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>> stopped.
>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>> shutdown complete.
>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>> amlauncher.ApplicationMasterLauncher
>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>> interrupted. Returning.
>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> NMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> AMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>> thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> AMLivelinessMonitor thread interrupted
>>>>>
>>>>> Has anyone seen this error before when building hadoop, and does
>>>>> anyone have any ideas?
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
I checked out source code from
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
builds.
>From TestMRJobsWithHistoryService.txt, the test passed.
What led to this test being singled out among the 454 tests ?
Thanks
On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> Yes well I tried 2.3, but I have found a number of problems building it. I
> had to resort to manually applying patches that I found in the bug tracking
> lists, which did not seem to have made it into all branches. So for the
> moment I am sticking with 2.2.0 which is advertised as being stable.
>
> I apologise for the confusion.
>
> Here is the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
> though perhaps not that illuminating:
>
>
>
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>
>
>
>
> On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
>
>> The output was
>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>
>> Can you show us the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> ?
>>
>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>> more attention to hadoop 2.3
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> I included the last part of
>>>
>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>
>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>
>>> Regards
>>>
>>> Chris
>>>
>>>
>>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>
>>>> What
>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> say ?
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>
>>>>> I am following the 'instructions' at
>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>>> as they are.
>>>>>
>>>>> I get the following test failure:
>>>>>
>>>>> Forking command line: /bin/sh -c cd
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>> &&
>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>
>>>>> Results :
>>>>>
>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Reactor Summary:
>>>>> [INFO]
>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>> [1:18:07.440s]
>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 1:18:07.828s
>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>> [INFO] Final Memory: 32M/94M
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [ERROR] Failed to execute goal
>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>> exception is java.util.concurrent.ExecutionException:
>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>
>>>>>
>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>
>>>>>
>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>> Applications still running : [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>> for Applications to be Finished
>>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from
>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>>> loghandler.NonAggregatingLogHandler
>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>> waiting for Applications to be Finished. Still alive:
>>>>> [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>>> unavailable on this system.
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>> is disabled.
>>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>>> localizer.ResourceLocalizationService
>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>> system...
>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>> stopped.
>>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>> shutdown complete.
>>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>>> amlauncher.ApplicationMasterLauncher
>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>> interrupted. Returning.
>>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> NMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> AMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>> thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>>> AMLivelinessMonitor thread interrupted
>>>>>
>>>>> Has anyone seen this error before when building hadoop, and does
>>>>> anyone have any ideas?
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
Yes well I tried 2.3, but I have found a number of problems building it. I
had to resort to manually applying patches that I found in the bug tracking
lists, which did not seem to have made it into all branches. So for the
moment I am sticking with 2.2.0 which is advertised as being stable.
I apologise for the confusion.
Here is the contents
of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
though perhaps not that illuminating:
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
> The output was
> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>
> Can you show us the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
> ?
>
> BTW hadoop 2.3 release candidate is coming up. You may consider paying
> more attention to hadoop 2.3
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I included the last part of
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService.txt
>>
>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>> Sloppy terminology I know, sorry if I wasn't very clear.
>>
>> Regards
>>
>> Chris
>>
>>
>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>
>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>
>>> What
>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>> say ?
>>>
>>> Cheers
>>>
>>>
>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>> on a Macbook Pro running OS X Mavericks.
>>>>
>>>> I am following the 'instructions' at
>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>> as they are.
>>>>
>>>> I get the following test failure:
>>>>
>>>> Forking command line: /bin/sh -c cd
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>> &&
>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>
>>>> Results :
>>>>
>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Reactor Summary:
>>>> [INFO]
>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>> [1:18:07.440s]
>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD FAILURE
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 1:18:07.828s
>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>> [INFO] Final Memory: 32M/94M
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>> exception is java.util.concurrent.ExecutionException:
>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>> properly goodbye. VM crash or System.exit called ?
>>>>
>>>>
>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>
>>>>
>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>> delegation.AbstractDelegationTokenSecretManager
>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>> sleep interrupted
>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>> Applications still running : [application_1391688017341_0001]
>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>> for Applications to be Finished
>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>> application_1391688017341_0001 transitioned from
>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>> loghandler.NonAggregatingLogHandler
>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>> waiting for Applications to be Finished. Still alive:
>>>> [application_1391688017341_0001]
>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>> unavailable on this system.
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>> is disabled.
>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>> localizer.ResourceLocalizationService
>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>> delegation.AbstractDelegationTokenSecretManager
>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>> sleep interrupted
>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>> system...
>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>> stopped.
>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>> shutdown complete.
>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>> amlauncher.ApplicationMasterLauncher
>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>> interrupted. Returning.
>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> NMLivelinessMonitor thread interrupted
>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>> Returning, interrupted : java.lang.InterruptedException
>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> AMLivelinessMonitor thread interrupted
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>> thread interrupted
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> AMLivelinessMonitor thread interrupted
>>>>
>>>> Has anyone seen this error before when building hadoop, and does anyone
>>>> have any ideas?
>>>>
>>>> Regards
>>>>
>>>> Chris
>>>>
>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
Yes well I tried 2.3, but I have found a number of problems building it. I
had to resort to manually applying patches that I found in the bug tracking
lists, which did not seem to have made it into all branches. So for the
moment I am sticking with 2.2.0 which is advertised as being stable.
I apologise for the confusion.
Here is the contents
of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
though perhaps not that illuminating:
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
> The output was
> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>
> Can you show us the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
> ?
>
> BTW hadoop 2.3 release candidate is coming up. You may consider paying
> more attention to hadoop 2.3
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I included the last part of
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService.txt
>>
>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>> Sloppy terminology I know, sorry if I wasn't very clear.
>>
>> Regards
>>
>> Chris
>>
>>
>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>
>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>
>>> What
>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>> say ?
>>>
>>> Cheers
>>>
>>>
>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>> on a Macbook Pro running OS X Mavericks.
>>>>
>>>> I am following the 'instructions' at
>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>> as they are.
>>>>
>>>> I get the following test failure:
>>>>
>>>> Forking command line: /bin/sh -c cd
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>> &&
>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>
>>>> Results :
>>>>
>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Reactor Summary:
>>>> [INFO]
>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>> [1:18:07.440s]
>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD FAILURE
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 1:18:07.828s
>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>> [INFO] Final Memory: 32M/94M
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>> exception is java.util.concurrent.ExecutionException:
>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>> properly goodbye. VM crash or System.exit called ?
>>>>
>>>>
>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>
>>>>
>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>> delegation.AbstractDelegationTokenSecretManager
>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>> sleep interrupted
>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>> Applications still running : [application_1391688017341_0001]
>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>> for Applications to be Finished
>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>> application_1391688017341_0001 transitioned from
>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>> loghandler.NonAggregatingLogHandler
>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>> waiting for Applications to be Finished. Still alive:
>>>> [application_1391688017341_0001]
>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>> unavailable on this system.
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>> is disabled.
>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>> localizer.ResourceLocalizationService
>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>> delegation.AbstractDelegationTokenSecretManager
>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>> sleep interrupted
>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>> system...
>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>> stopped.
>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>> shutdown complete.
>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>> amlauncher.ApplicationMasterLauncher
>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>> interrupted. Returning.
>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> NMLivelinessMonitor thread interrupted
>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>> Returning, interrupted : java.lang.InterruptedException
>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> AMLivelinessMonitor thread interrupted
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>> thread interrupted
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> AMLivelinessMonitor thread interrupted
>>>>
>>>> Has anyone seen this error before when building hadoop, and does anyone
>>>> have any ideas?
>>>>
>>>> Regards
>>>>
>>>> Chris
>>>>
>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
Yes well I tried 2.3, but I have found a number of problems building it. I
had to resort to manually applying patches that I found in the bug tracking
lists, which did not seem to have made it into all branches. So for the
moment I am sticking with 2.2.0 which is advertised as being stable.
I apologise for the confusion.
Here is the contents
of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
though perhaps not that illuminating:
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
> The output was
> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>
> Can you show us the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
> ?
>
> BTW hadoop 2.3 release candidate is coming up. You may consider paying
> more attention to hadoop 2.3
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I included the last part of
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService.txt
>>
>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>> Sloppy terminology I know, sorry if I wasn't very clear.
>>
>> Regards
>>
>> Chris
>>
>>
>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>
>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>
>>> What
>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>> say ?
>>>
>>> Cheers
>>>
>>>
>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>> on a Macbook Pro running OS X Mavericks.
>>>>
>>>> I am following the 'instructions' at
>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>> as they are.
>>>>
>>>> I get the following test failure:
>>>>
>>>> Forking command line: /bin/sh -c cd
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>> &&
>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>
>>>> Results :
>>>>
>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Reactor Summary:
>>>> [INFO]
>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>> [1:18:07.440s]
>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD FAILURE
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 1:18:07.828s
>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>> [INFO] Final Memory: 32M/94M
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>> exception is java.util.concurrent.ExecutionException:
>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>> properly goodbye. VM crash or System.exit called ?
>>>>
>>>>
>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>
>>>>
>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>> delegation.AbstractDelegationTokenSecretManager
>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>> sleep interrupted
>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>> Applications still running : [application_1391688017341_0001]
>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>> for Applications to be Finished
>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>> application_1391688017341_0001 transitioned from
>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>> loghandler.NonAggregatingLogHandler
>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>> waiting for Applications to be Finished. Still alive:
>>>> [application_1391688017341_0001]
>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>> unavailable on this system.
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>> is disabled.
>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>> localizer.ResourceLocalizationService
>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>> delegation.AbstractDelegationTokenSecretManager
>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>> sleep interrupted
>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>> system...
>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>> stopped.
>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>> shutdown complete.
>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>> amlauncher.ApplicationMasterLauncher
>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>> interrupted. Returning.
>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> NMLivelinessMonitor thread interrupted
>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>> Returning, interrupted : java.lang.InterruptedException
>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> AMLivelinessMonitor thread interrupted
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>> thread interrupted
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> AMLivelinessMonitor thread interrupted
>>>>
>>>> Has anyone seen this error before when building hadoop, and does anyone
>>>> have any ideas?
>>>>
>>>> Regards
>>>>
>>>> Chris
>>>>
>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
Yes well I tried 2.3, but I have found a number of problems building it. I
had to resort to manually applying patches that I found in the bug tracking
lists, which did not seem to have made it into all branches. So for the
moment I am sticking with 2.2.0 which is advertised as being stable.
I apologise for the confusion.
Here is the contents
of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
though perhaps not that illuminating:
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
On 7 February 2014 14:12, Ted Yu <yu...@gmail.com> wrote:
> The output was
> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>
> Can you show us the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
> ?
>
> BTW hadoop 2.3 release candidate is coming up. You may consider paying
> more attention to hadoop 2.3
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I included the last part of
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService.txt
>>
>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>> Sloppy terminology I know, sorry if I wasn't very clear.
>>
>> Regards
>>
>> Chris
>>
>>
>> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>>
>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>
>>> What
>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>> say ?
>>>
>>> Cheers
>>>
>>>
>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>> on a Macbook Pro running OS X Mavericks.
>>>>
>>>> I am following the 'instructions' at
>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>>> as they are.
>>>>
>>>> I get the following test failure:
>>>>
>>>> Forking command line: /bin/sh -c cd
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>> &&
>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>
>>>> Results :
>>>>
>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Reactor Summary:
>>>> [INFO]
>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>> [1:18:07.440s]
>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD FAILURE
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 1:18:07.828s
>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>> [INFO] Final Memory: 32M/94M
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>> exception is java.util.concurrent.ExecutionException:
>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>> properly goodbye. VM crash or System.exit called ?
>>>>
>>>>
>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>
>>>>
>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>> delegation.AbstractDelegationTokenSecretManager
>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>> sleep interrupted
>>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:00:54,759 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>> Applications still running : [application_1391688017341_0001]
>>>> 2014-02-06 23:00:54,762 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>> for Applications to be Finished
>>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>> application_1391688017341_0001 transitioned from
>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>>> loghandler.NonAggregatingLogHandler
>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>>> nodemanager.DefaultContainerExecutor
>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>>> :
>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>> 2014-02-06 23:00:58,766 INFO [main]
>>>> containermanager.ContainerManagerImpl
>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>> waiting for Applications to be Finished. Still alive:
>>>> [application_1391688017341_0001]
>>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>>> unavailable on this system.
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>> is disabled.
>>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>>> localizer.ResourceLocalizationService
>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>>> (Slf4jLog.java:info(67)) - Stopped
>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>> delegation.AbstractDelegationTokenSecretManager
>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>> sleep interrupted
>>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>> system...
>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>> stopped.
>>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>> shutdown complete.
>>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>>> amlauncher.ApplicationMasterLauncher
>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>> interrupted. Returning.
>>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server
>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>>>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> NMLivelinessMonitor thread interrupted
>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>> Returning, interrupted : java.lang.InterruptedException
>>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> AMLivelinessMonitor thread interrupted
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>> thread interrupted
>>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>>> AMLivelinessMonitor thread interrupted
>>>>
>>>> Has anyone seen this error before when building hadoop, and does anyone
>>>> have any ideas?
>>>>
>>>> Regards
>>>>
>>>> Chris
>>>>
>>>
>>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
The output was
from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
Can you show us the contents
of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
?
BTW hadoop 2.3 release candidate is coming up. You may consider paying more
attention to hadoop 2.3
Cheers
On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> I included the last part of
>
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
> TestMRJobsWithHistoryService.txt
>
> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
> Sloppy terminology I know, sorry if I wasn't very clear.
>
> Regards
>
> Chris
>
>
> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>
>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>
>> What
>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> say ?
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on
>>> a Macbook Pro running OS X Mavericks.
>>>
>>> I am following the 'instructions' at
>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>> as they are.
>>>
>>> I get the following test failure:
>>>
>>> Forking command line: /bin/sh -c cd
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>> &&
>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>
>>> Results :
>>>
>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>> [1:18:07.440s]
>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 1:18:07.828s
>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>> [INFO] Final Memory: 32M/94M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>> exception is java.util.concurrent.ExecutionException:
>>> java.lang.RuntimeException: The forked VM terminated without saying
>>> properly goodbye. VM crash or System.exit called ?
>>>
>>>
>>> and the output from TestMRJobsWithHistoryService includes:
>>>
>>>
>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>> delegation.AbstractDelegationTokenSecretManager
>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>> sleep interrupted
>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55131
>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:00:54,759 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>> Applications still running : [application_1391688017341_0001]
>>> 2014-02-06 23:00:54,762 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>> for Applications to be Finished
>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>> application_1391688017341_0001 transitioned from RUNNING to
>>> APPLICATION_RESOURCES_CLEANINGUP
>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>> APPLICATION_STOP for appId application_1391688017341_0001
>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>> application_1391688017341_0001 transitioned from
>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>> loghandler.NonAggregatingLogHandler
>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:58,766 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>> waiting for Applications to be Finished. Still alive:
>>> [application_1391688017341_0001]
>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55125
>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>> unavailable on this system.
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>> is disabled.
>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55126
>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>> localizer.ResourceLocalizationService
>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>> delegation.AbstractDelegationTokenSecretManager
>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>> sleep interrupted
>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>> system...
>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>> stopped.
>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>> shutdown complete.
>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>> amlauncher.ApplicationMasterLauncher
>>> (ApplicationMasterLauncher.java:run(98)) -
>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>> interrupted. Returning.
>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55122
>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55121
>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55120
>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55119
>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> NMLivelinessMonitor thread interrupted
>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>> Returning, interrupted : java.lang.InterruptedException
>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> AMLivelinessMonitor thread interrupted
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>> thread interrupted
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> AMLivelinessMonitor thread interrupted
>>>
>>> Has anyone seen this error before when building hadoop, and does anyone
>>> have any ideas?
>>>
>>> Regards
>>>
>>> Chris
>>>
>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
The output was
from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
Can you show us the contents
of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
?
BTW hadoop 2.3 release candidate is coming up. You may consider paying more
attention to hadoop 2.3
Cheers
On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> I included the last part of
>
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
> TestMRJobsWithHistoryService.txt
>
> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
> Sloppy terminology I know, sorry if I wasn't very clear.
>
> Regards
>
> Chris
>
>
> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>
>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>
>> What
>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> say ?
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on
>>> a Macbook Pro running OS X Mavericks.
>>>
>>> I am following the 'instructions' at
>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>> as they are.
>>>
>>> I get the following test failure:
>>>
>>> Forking command line: /bin/sh -c cd
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>> &&
>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>
>>> Results :
>>>
>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>> [1:18:07.440s]
>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 1:18:07.828s
>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>> [INFO] Final Memory: 32M/94M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>> exception is java.util.concurrent.ExecutionException:
>>> java.lang.RuntimeException: The forked VM terminated without saying
>>> properly goodbye. VM crash or System.exit called ?
>>>
>>>
>>> and the output from TestMRJobsWithHistoryService includes:
>>>
>>>
>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>> delegation.AbstractDelegationTokenSecretManager
>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>> sleep interrupted
>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55131
>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:00:54,759 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>> Applications still running : [application_1391688017341_0001]
>>> 2014-02-06 23:00:54,762 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>> for Applications to be Finished
>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>> application_1391688017341_0001 transitioned from RUNNING to
>>> APPLICATION_RESOURCES_CLEANINGUP
>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>> APPLICATION_STOP for appId application_1391688017341_0001
>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>> application_1391688017341_0001 transitioned from
>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>> loghandler.NonAggregatingLogHandler
>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:58,766 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>> waiting for Applications to be Finished. Still alive:
>>> [application_1391688017341_0001]
>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55125
>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>> unavailable on this system.
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>> is disabled.
>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55126
>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>> localizer.ResourceLocalizationService
>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>> delegation.AbstractDelegationTokenSecretManager
>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>> sleep interrupted
>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>> system...
>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>> stopped.
>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>> shutdown complete.
>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>> amlauncher.ApplicationMasterLauncher
>>> (ApplicationMasterLauncher.java:run(98)) -
>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>> interrupted. Returning.
>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55122
>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55121
>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55120
>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55119
>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> NMLivelinessMonitor thread interrupted
>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>> Returning, interrupted : java.lang.InterruptedException
>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> AMLivelinessMonitor thread interrupted
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>> thread interrupted
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> AMLivelinessMonitor thread interrupted
>>>
>>> Has anyone seen this error before when building hadoop, and does anyone
>>> have any ideas?
>>>
>>> Regards
>>>
>>> Chris
>>>
>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
The output was
from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
Can you show us the contents
of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
?
BTW hadoop 2.3 release candidate is coming up. You may consider paying more
attention to hadoop 2.3
Cheers
On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> I included the last part of
>
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
> TestMRJobsWithHistoryService.txt
>
> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
> Sloppy terminology I know, sorry if I wasn't very clear.
>
> Regards
>
> Chris
>
>
> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>
>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>
>> What
>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> say ?
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on
>>> a Macbook Pro running OS X Mavericks.
>>>
>>> I am following the 'instructions' at
>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>> as they are.
>>>
>>> I get the following test failure:
>>>
>>> Forking command line: /bin/sh -c cd
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>> &&
>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>
>>> Results :
>>>
>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>> [1:18:07.440s]
>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 1:18:07.828s
>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>> [INFO] Final Memory: 32M/94M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>> exception is java.util.concurrent.ExecutionException:
>>> java.lang.RuntimeException: The forked VM terminated without saying
>>> properly goodbye. VM crash or System.exit called ?
>>>
>>>
>>> and the output from TestMRJobsWithHistoryService includes:
>>>
>>>
>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>> delegation.AbstractDelegationTokenSecretManager
>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>> sleep interrupted
>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55131
>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:00:54,759 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>> Applications still running : [application_1391688017341_0001]
>>> 2014-02-06 23:00:54,762 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>> for Applications to be Finished
>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>> application_1391688017341_0001 transitioned from RUNNING to
>>> APPLICATION_RESOURCES_CLEANINGUP
>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>> APPLICATION_STOP for appId application_1391688017341_0001
>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>> application_1391688017341_0001 transitioned from
>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>> loghandler.NonAggregatingLogHandler
>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:58,766 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>> waiting for Applications to be Finished. Still alive:
>>> [application_1391688017341_0001]
>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55125
>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>> unavailable on this system.
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>> is disabled.
>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55126
>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>> localizer.ResourceLocalizationService
>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>> delegation.AbstractDelegationTokenSecretManager
>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>> sleep interrupted
>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>> system...
>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>> stopped.
>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>> shutdown complete.
>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>> amlauncher.ApplicationMasterLauncher
>>> (ApplicationMasterLauncher.java:run(98)) -
>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>> interrupted. Returning.
>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55122
>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55121
>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55120
>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55119
>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> NMLivelinessMonitor thread interrupted
>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>> Returning, interrupted : java.lang.InterruptedException
>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> AMLivelinessMonitor thread interrupted
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>> thread interrupted
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> AMLivelinessMonitor thread interrupted
>>>
>>> Has anyone seen this error before when building hadoop, and does anyone
>>> have any ideas?
>>>
>>> Regards
>>>
>>> Chris
>>>
>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
The output was
from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
Can you show us the contents
of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
?
BTW hadoop 2.3 release candidate is coming up. You may consider paying more
attention to hadoop 2.3
Cheers
On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> I included the last part of
>
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
> TestMRJobsWithHistoryService.txt
>
> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
> Sloppy terminology I know, sorry if I wasn't very clear.
>
> Regards
>
> Chris
>
>
> On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
>
>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>
>> What
>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> say ?
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on
>>> a Macbook Pro running OS X Mavericks.
>>>
>>> I am following the 'instructions' at
>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>>> as they are.
>>>
>>> I get the following test failure:
>>>
>>> Forking command line: /bin/sh -c cd
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>> &&
>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>
>>> Results :
>>>
>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>> [1:18:07.440s]
>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 1:18:07.828s
>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>> [INFO] Final Memory: 32M/94M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>> exception is java.util.concurrent.ExecutionException:
>>> java.lang.RuntimeException: The forked VM terminated without saying
>>> properly goodbye. VM crash or System.exit called ?
>>>
>>>
>>> and the output from TestMRJobsWithHistoryService includes:
>>>
>>>
>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>> delegation.AbstractDelegationTokenSecretManager
>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>> sleep interrupted
>>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55131
>>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:00:54,759 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>> Applications still running : [application_1391688017341_0001]
>>> 2014-02-06 23:00:54,762 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>> for Applications to be Finished
>>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>> application_1391688017341_0001 transitioned from RUNNING to
>>> APPLICATION_RESOURCES_CLEANINGUP
>>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>> APPLICATION_STOP for appId application_1391688017341_0001
>>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>> application_1391688017341_0001 transitioned from
>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>>> loghandler.NonAggregatingLogHandler
>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>>> nodemanager.DefaultContainerExecutor
>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>>> :
>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>> 2014-02-06 23:00:58,766 INFO [main]
>>> containermanager.ContainerManagerImpl
>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>> waiting for Applications to be Finished. Still alive:
>>> [application_1391688017341_0001]
>>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55125
>>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>>> unavailable on this system.
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>> is disabled.
>>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55126
>>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>>> localizer.ResourceLocalizationService
>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log
>>> (Slf4jLog.java:info(67)) - Stopped
>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>> delegation.AbstractDelegationTokenSecretManager
>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>> sleep interrupted
>>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>> system...
>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>> stopped.
>>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>> shutdown complete.
>>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>>> amlauncher.ApplicationMasterLauncher
>>> (ApplicationMasterLauncher.java:run(98)) -
>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>> interrupted. Returning.
>>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55122
>>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55121
>>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55120
>>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437))
>>> - Stopping server on 55119
>>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> NMLivelinessMonitor thread interrupted
>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>> Returning, interrupted : java.lang.InterruptedException
>>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> AMLivelinessMonitor thread interrupted
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>> thread interrupted
>>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>>> AMLivelinessMonitor thread interrupted
>>>
>>> Has anyone seen this error before when building hadoop, and does anyone
>>> have any ideas?
>>>
>>> Regards
>>>
>>> Chris
>>>
>>
>>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
I included the last part of
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
TestMRJobsWithHistoryService.txt
in the second half of my initial posting, calling it "the output from
TestMRJobsWithHistoryService".
Sloppy terminology I know, sorry if I wasn't very clear.
Regards
Chris
On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
> There isn't System.exit call in TestMRJobsWithHistoryService.java
>
> What
> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
> say ?
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> Hi,
>>
>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on
>> a Macbook Pro running OS X Mavericks.
>>
>> I am following the 'instructions' at
>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>> as they are.
>>
>> I get the following test failure:
>>
>> Forking command line: /bin/sh -c cd
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>> &&
>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>
>> Results :
>>
>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>> [1:18:07.440s]
>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>> [INFO] hadoop-mapreduce .................................. SKIPPED
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 1:18:07.828s
>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>> [INFO] Final Memory: 32M/94M
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>> exception is java.util.concurrent.ExecutionException:
>> java.lang.RuntimeException: The forked VM terminated without saying
>> properly goodbye. VM crash or System.exit called ?
>>
>>
>> and the output from TestMRJobsWithHistoryService includes:
>>
>>
>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>> delegation.AbstractDelegationTokenSecretManager
>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>> sleep interrupted
>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55131
>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:00:54,759 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>> Applications still running : [application_1391688017341_0001]
>> 2014-02-06 23:00:54,762 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>> for Applications to be Finished
>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>> application.Application (ApplicationImpl.java:handle(451)) - Application
>> application_1391688017341_0001 transitioned from RUNNING to
>> APPLICATION_RESOURCES_CLEANINGUP
>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>> APPLICATION_STOP for appId application_1391688017341_0001
>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>> application.Application (ApplicationImpl.java:handle(451)) - Application
>> application_1391688017341_0001 transitioned from
>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>> loghandler.NonAggregatingLogHandler
>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>> application: application_1391688017341_0001, with delay of 10800 seconds
>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:58,766 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>> waiting for Applications to be Finished. Still alive:
>> [application_1391688017341_0001]
>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55125
>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>> unavailable on this system.
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> is disabled.
>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55126
>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>> localizer.ResourceLocalizationService
>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>> delegation.AbstractDelegationTokenSecretManager
>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>> sleep interrupted
>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>> system...
>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>> stopped.
>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>> shutdown complete.
>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>> amlauncher.ApplicationMasterLauncher
>> (ApplicationMasterLauncher.java:run(98)) -
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>> interrupted. Returning.
>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55122
>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55121
>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55120
>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55119
>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> NMLivelinessMonitor thread interrupted
>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>> Returning, interrupted : java.lang.InterruptedException
>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> AMLivelinessMonitor thread interrupted
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>> thread interrupted
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> AMLivelinessMonitor thread interrupted
>>
>> Has anyone seen this error before when building hadoop, and does anyone
>> have any ideas?
>>
>> Regards
>>
>> Chris
>>
>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
I included the last part of
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
TestMRJobsWithHistoryService.txt
in the second half of my initial posting, calling it "the output from
TestMRJobsWithHistoryService".
Sloppy terminology I know, sorry if I wasn't very clear.
Regards
Chris
On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
> There isn't System.exit call in TestMRJobsWithHistoryService.java
>
> What
> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
> say ?
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> Hi,
>>
>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on
>> a Macbook Pro running OS X Mavericks.
>>
>> I am following the 'instructions' at
>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>> as they are.
>>
>> I get the following test failure:
>>
>> Forking command line: /bin/sh -c cd
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>> &&
>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>
>> Results :
>>
>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>> [1:18:07.440s]
>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>> [INFO] hadoop-mapreduce .................................. SKIPPED
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 1:18:07.828s
>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>> [INFO] Final Memory: 32M/94M
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>> exception is java.util.concurrent.ExecutionException:
>> java.lang.RuntimeException: The forked VM terminated without saying
>> properly goodbye. VM crash or System.exit called ?
>>
>>
>> and the output from TestMRJobsWithHistoryService includes:
>>
>>
>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>> delegation.AbstractDelegationTokenSecretManager
>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>> sleep interrupted
>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55131
>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:00:54,759 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>> Applications still running : [application_1391688017341_0001]
>> 2014-02-06 23:00:54,762 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>> for Applications to be Finished
>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>> application.Application (ApplicationImpl.java:handle(451)) - Application
>> application_1391688017341_0001 transitioned from RUNNING to
>> APPLICATION_RESOURCES_CLEANINGUP
>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>> APPLICATION_STOP for appId application_1391688017341_0001
>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>> application.Application (ApplicationImpl.java:handle(451)) - Application
>> application_1391688017341_0001 transitioned from
>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>> loghandler.NonAggregatingLogHandler
>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>> application: application_1391688017341_0001, with delay of 10800 seconds
>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:58,766 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>> waiting for Applications to be Finished. Still alive:
>> [application_1391688017341_0001]
>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55125
>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>> unavailable on this system.
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> is disabled.
>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55126
>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>> localizer.ResourceLocalizationService
>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>> delegation.AbstractDelegationTokenSecretManager
>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>> sleep interrupted
>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>> system...
>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>> stopped.
>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>> shutdown complete.
>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>> amlauncher.ApplicationMasterLauncher
>> (ApplicationMasterLauncher.java:run(98)) -
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>> interrupted. Returning.
>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55122
>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55121
>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55120
>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55119
>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> NMLivelinessMonitor thread interrupted
>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>> Returning, interrupted : java.lang.InterruptedException
>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> AMLivelinessMonitor thread interrupted
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>> thread interrupted
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> AMLivelinessMonitor thread interrupted
>>
>> Has anyone seen this error before when building hadoop, and does anyone
>> have any ideas?
>>
>> Regards
>>
>> Chris
>>
>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
I included the last part of
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
TestMRJobsWithHistoryService.txt
in the second half of my initial posting, calling it "the output from
TestMRJobsWithHistoryService".
Sloppy terminology I know, sorry if I wasn't very clear.
Regards
Chris
On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
> There isn't System.exit call in TestMRJobsWithHistoryService.java
>
> What
> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
> say ?
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> Hi,
>>
>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on
>> a Macbook Pro running OS X Mavericks.
>>
>> I am following the 'instructions' at
>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>> as they are.
>>
>> I get the following test failure:
>>
>> Forking command line: /bin/sh -c cd
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>> &&
>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>
>> Results :
>>
>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>> [1:18:07.440s]
>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>> [INFO] hadoop-mapreduce .................................. SKIPPED
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 1:18:07.828s
>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>> [INFO] Final Memory: 32M/94M
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>> exception is java.util.concurrent.ExecutionException:
>> java.lang.RuntimeException: The forked VM terminated without saying
>> properly goodbye. VM crash or System.exit called ?
>>
>>
>> and the output from TestMRJobsWithHistoryService includes:
>>
>>
>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>> delegation.AbstractDelegationTokenSecretManager
>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>> sleep interrupted
>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55131
>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:00:54,759 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>> Applications still running : [application_1391688017341_0001]
>> 2014-02-06 23:00:54,762 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>> for Applications to be Finished
>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>> application.Application (ApplicationImpl.java:handle(451)) - Application
>> application_1391688017341_0001 transitioned from RUNNING to
>> APPLICATION_RESOURCES_CLEANINGUP
>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>> APPLICATION_STOP for appId application_1391688017341_0001
>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>> application.Application (ApplicationImpl.java:handle(451)) - Application
>> application_1391688017341_0001 transitioned from
>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>> loghandler.NonAggregatingLogHandler
>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>> application: application_1391688017341_0001, with delay of 10800 seconds
>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:58,766 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>> waiting for Applications to be Finished. Still alive:
>> [application_1391688017341_0001]
>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55125
>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>> unavailable on this system.
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> is disabled.
>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55126
>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>> localizer.ResourceLocalizationService
>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>> delegation.AbstractDelegationTokenSecretManager
>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>> sleep interrupted
>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>> system...
>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>> stopped.
>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>> shutdown complete.
>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>> amlauncher.ApplicationMasterLauncher
>> (ApplicationMasterLauncher.java:run(98)) -
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>> interrupted. Returning.
>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55122
>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55121
>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55120
>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55119
>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> NMLivelinessMonitor thread interrupted
>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>> Returning, interrupted : java.lang.InterruptedException
>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> AMLivelinessMonitor thread interrupted
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>> thread interrupted
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> AMLivelinessMonitor thread interrupted
>>
>> Has anyone seen this error before when building hadoop, and does anyone
>> have any ideas?
>>
>> Regards
>>
>> Chris
>>
>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Christopher Thomas <ch...@gmail.com>.
I included the last part of
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
TestMRJobsWithHistoryService.txt
in the second half of my initial posting, calling it "the output from
TestMRJobsWithHistoryService".
Sloppy terminology I know, sorry if I wasn't very clear.
Regards
Chris
On 7 February 2014 11:53, Ted Yu <yu...@gmail.com> wrote:
> There isn't System.exit call in TestMRJobsWithHistoryService.java
>
> What
> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
> say ?
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> Hi,
>>
>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on
>> a Macbook Pro running OS X Mavericks.
>>
>> I am following the 'instructions' at
>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
>> as they are.
>>
>> I get the following test failure:
>>
>> Forking command line: /bin/sh -c cd
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>> &&
>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>
>> Results :
>>
>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>> [1:18:07.440s]
>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>> [INFO] hadoop-mapreduce .................................. SKIPPED
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 1:18:07.828s
>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>> [INFO] Final Memory: 32M/94M
>> [INFO]
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>> exception is java.util.concurrent.ExecutionException:
>> java.lang.RuntimeException: The forked VM terminated without saying
>> properly goodbye. VM crash or System.exit called ?
>>
>>
>> and the output from TestMRJobsWithHistoryService includes:
>>
>>
>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>> delegation.AbstractDelegationTokenSecretManager
>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>> sleep interrupted
>> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55131
>> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55131
>> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
>> 2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:00:54,759 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>> Applications still running : [application_1391688017341_0001]
>> 2014-02-06 23:00:54,762 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>> for Applications to be Finished
>> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
>> application.Application (ApplicationImpl.java:handle(451)) - Application
>> application_1391688017341_0001 transitioned from RUNNING to
>> APPLICATION_RESOURCES_CLEANINGUP
>> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>> APPLICATION_STOP for appId application_1391688017341_0001
>> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
>> application.Application (ApplicationImpl.java:handle(451)) - Application
>> application_1391688017341_0001 transitioned from
>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
>> loghandler.NonAggregatingLogHandler
>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
>> application: application_1391688017341_0001, with delay of 10800 seconds
>> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
>> nodemanager.DefaultContainerExecutor
>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
>> :
>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>> 2014-02-06 23:00:58,766 INFO [main]
>> containermanager.ContainerManagerImpl
>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>> waiting for Applications to be Finished. Still alive:
>> [application_1391688017341_0001]
>> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55125
>> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55125
>> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
>> unavailable on this system.
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>> is disabled.
>> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55126
>> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55126
>> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,785 INFO [Public Localizer]
>> localizer.ResourceLocalizationService
>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>> 2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67))
>> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>> delegation.AbstractDelegationTokenSecretManager
>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>> sleep interrupted
>> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>> system...
>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>> stopped.
>> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>> shutdown complete.
>> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
>> amlauncher.ApplicationMasterLauncher
>> (ApplicationMasterLauncher.java:run(98)) -
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>> interrupted. Returning.
>> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55122
>> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55122
>> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55121
>> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55121
>> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55120
>> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55120
>> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437))
>> - Stopping server on 55119
>> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
>> (Server.java:run(720)) - Stopping IPC Server listener on 55119
>> 2014-02-06 23:01:08,917 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> NMLivelinessMonitor thread interrupted
>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>> Returning, interrupted : java.lang.InterruptedException
>> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
>> (Server.java:run(866)) - Stopping IPC Server Responder
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> AMLivelinessMonitor thread interrupted
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>> thread interrupted
>> 2014-02-06 23:01:08,918 INFO [Ping Checker]
>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
>> AMLivelinessMonitor thread interrupted
>>
>> Has anyone seen this error before when building hadoop, and does anyone
>> have any ideas?
>>
>> Regards
>>
>> Chris
>>
>
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
There isn't System.exit call in TestMRJobsWithHistoryService.java
What
did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
say ?
Cheers
On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> Hi,
>
> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on a
> Macbook Pro running OS X Mavericks.
>
> I am following the 'instructions' at
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
> as they are.
>
> I get the following test failure:
>
> Forking command line: /bin/sh -c cd
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
> &&
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>
> Results :
>
> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
> [1:18:07.440s]
> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
> [INFO] hadoop-mapreduce .................................. SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1:18:07.828s
> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
> [INFO] Final Memory: 32M/94M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
> exception is java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: The forked VM terminated without saying
> properly goodbye. VM crash or System.exit called ?
>
>
> and the output from TestMRJobsWithHistoryService includes:
>
>
> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
> sleep interrupted
> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55131
> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55131
> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
> 2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:00:54,759 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
> Applications still running : [application_1391688017341_0001]
> 2014-02-06 23:00:54,762 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
> for Applications to be Finished
> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
> application.Application (ApplicationImpl.java:handle(451)) - Application
> application_1391688017341_0001 transitioned from RUNNING to
> APPLICATION_RESOURCES_CLEANINGUP
> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
> APPLICATION_STOP for appId application_1391688017341_0001
> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
> application.Application (ApplicationImpl.java:handle(451)) - Application
> application_1391688017341_0001 transitioned from
> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
> loghandler.NonAggregatingLogHandler
> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
> application: application_1391688017341_0001, with delay of 10800 seconds
> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:58,766 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
> waiting for Applications to be Finished. Still alive:
> [application_1391688017341_0001]
> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55125
> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55125
> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
> unavailable on this system.
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> is disabled.
> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55126
> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55126
> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,785 INFO [Public Localizer]
> localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:run(728)) - Public cache exiting
> 2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
> sleep interrupted
> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
> system...
> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
> stopped.
> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
> shutdown complete.
> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
> amlauncher.ApplicationMasterLauncher
> (ApplicationMasterLauncher.java:run(98)) -
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
> interrupted. Returning.
> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55122
> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55122
> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55121
> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55121
> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55120
> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55120
> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55119
> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55119
> 2014-02-06 23:01:08,917 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> NMLivelinessMonitor thread interrupted
> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
> Returning, interrupted : java.lang.InterruptedException
> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> AMLivelinessMonitor thread interrupted
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
> thread interrupted
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> AMLivelinessMonitor thread interrupted
>
> Has anyone seen this error before when building hadoop, and does anyone
> have any ideas?
>
> Regards
>
> Chris
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
There isn't System.exit call in TestMRJobsWithHistoryService.java
What
did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
say ?
Cheers
On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> Hi,
>
> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on a
> Macbook Pro running OS X Mavericks.
>
> I am following the 'instructions' at
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
> as they are.
>
> I get the following test failure:
>
> Forking command line: /bin/sh -c cd
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
> &&
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>
> Results :
>
> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
> [1:18:07.440s]
> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
> [INFO] hadoop-mapreduce .................................. SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1:18:07.828s
> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
> [INFO] Final Memory: 32M/94M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
> exception is java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: The forked VM terminated without saying
> properly goodbye. VM crash or System.exit called ?
>
>
> and the output from TestMRJobsWithHistoryService includes:
>
>
> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
> sleep interrupted
> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55131
> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55131
> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
> 2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:00:54,759 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
> Applications still running : [application_1391688017341_0001]
> 2014-02-06 23:00:54,762 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
> for Applications to be Finished
> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
> application.Application (ApplicationImpl.java:handle(451)) - Application
> application_1391688017341_0001 transitioned from RUNNING to
> APPLICATION_RESOURCES_CLEANINGUP
> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
> APPLICATION_STOP for appId application_1391688017341_0001
> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
> application.Application (ApplicationImpl.java:handle(451)) - Application
> application_1391688017341_0001 transitioned from
> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
> loghandler.NonAggregatingLogHandler
> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
> application: application_1391688017341_0001, with delay of 10800 seconds
> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:58,766 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
> waiting for Applications to be Finished. Still alive:
> [application_1391688017341_0001]
> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55125
> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55125
> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
> unavailable on this system.
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> is disabled.
> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55126
> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55126
> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,785 INFO [Public Localizer]
> localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:run(728)) - Public cache exiting
> 2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
> sleep interrupted
> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
> system...
> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
> stopped.
> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
> shutdown complete.
> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
> amlauncher.ApplicationMasterLauncher
> (ApplicationMasterLauncher.java:run(98)) -
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
> interrupted. Returning.
> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55122
> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55122
> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55121
> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55121
> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55120
> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55120
> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55119
> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55119
> 2014-02-06 23:01:08,917 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> NMLivelinessMonitor thread interrupted
> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
> Returning, interrupted : java.lang.InterruptedException
> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> AMLivelinessMonitor thread interrupted
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
> thread interrupted
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> AMLivelinessMonitor thread interrupted
>
> Has anyone seen this error before when building hadoop, and does anyone
> have any ideas?
>
> Regards
>
> Chris
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
There isn't System.exit call in TestMRJobsWithHistoryService.java
What
did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
say ?
Cheers
On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> Hi,
>
> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on a
> Macbook Pro running OS X Mavericks.
>
> I am following the 'instructions' at
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
> as they are.
>
> I get the following test failure:
>
> Forking command line: /bin/sh -c cd
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
> &&
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>
> Results :
>
> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
> [1:18:07.440s]
> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
> [INFO] hadoop-mapreduce .................................. SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1:18:07.828s
> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
> [INFO] Final Memory: 32M/94M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
> exception is java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: The forked VM terminated without saying
> properly goodbye. VM crash or System.exit called ?
>
>
> and the output from TestMRJobsWithHistoryService includes:
>
>
> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
> sleep interrupted
> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55131
> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55131
> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
> 2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:00:54,759 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
> Applications still running : [application_1391688017341_0001]
> 2014-02-06 23:00:54,762 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
> for Applications to be Finished
> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
> application.Application (ApplicationImpl.java:handle(451)) - Application
> application_1391688017341_0001 transitioned from RUNNING to
> APPLICATION_RESOURCES_CLEANINGUP
> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
> APPLICATION_STOP for appId application_1391688017341_0001
> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
> application.Application (ApplicationImpl.java:handle(451)) - Application
> application_1391688017341_0001 transitioned from
> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
> loghandler.NonAggregatingLogHandler
> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
> application: application_1391688017341_0001, with delay of 10800 seconds
> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:58,766 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
> waiting for Applications to be Finished. Still alive:
> [application_1391688017341_0001]
> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55125
> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55125
> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
> unavailable on this system.
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> is disabled.
> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55126
> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55126
> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,785 INFO [Public Localizer]
> localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:run(728)) - Public cache exiting
> 2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
> sleep interrupted
> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
> system...
> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
> stopped.
> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
> shutdown complete.
> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
> amlauncher.ApplicationMasterLauncher
> (ApplicationMasterLauncher.java:run(98)) -
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
> interrupted. Returning.
> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55122
> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55122
> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55121
> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55121
> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55120
> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55120
> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55119
> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55119
> 2014-02-06 23:01:08,917 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> NMLivelinessMonitor thread interrupted
> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
> Returning, interrupted : java.lang.InterruptedException
> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> AMLivelinessMonitor thread interrupted
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
> thread interrupted
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> AMLivelinessMonitor thread interrupted
>
> Has anyone seen this error before when building hadoop, and does anyone
> have any ideas?
>
> Regards
>
> Chris
>
Re: Problems building hadoop 2.2.0 from source
Posted by Ted Yu <yu...@gmail.com>.
There isn't System.exit call in TestMRJobsWithHistoryService.java
What
did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
say ?
Cheers
On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:
> Hi,
>
> I am a complete beginner to Hadoop, trying to build 2.2.0 from source on a
> Macbook Pro running OS X Mavericks.
>
> I am following the 'instructions' at
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html such
> as they are.
>
> I get the following test failure:
>
> Forking command line: /bin/sh -c cd
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
> &&
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>
> Results :
>
> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
> [1:18:07.440s]
> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
> [INFO] hadoop-mapreduce .................................. SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1:18:07.828s
> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
> [INFO] Final Memory: 32M/94M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
> exception is java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: The forked VM terminated without saying
> properly goodbye. VM crash or System.exit called ?
>
>
> and the output from TestMRJobsWithHistoryService includes:
>
>
> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
> sleep interrupted
> 2014-02-06 23:00:54,531 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55131
> 2014-02-06 23:00:54,540 INFO [IPC Server listener on 55131] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55131
> 2014-02-06 23:00:54,540 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:00:54,544 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
> 2014-02-06 23:00:54,647 INFO [main] hs.JobHistory
> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To Done
> 2014-02-06 23:00:54,656 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:00:54,759 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
> Applications still running : [application_1391688017341_0001]
> 2014-02-06 23:00:54,762 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
> for Applications to be Finished
> 2014-02-06 23:00:54,762 INFO [AsyncDispatcher event handler]
> application.Application (ApplicationImpl.java:handle(451)) - Application
> application_1391688017341_0001 transitioned from RUNNING to
> APPLICATION_RESOURCES_CLEANINGUP
> 2014-02-06 23:00:54,763 INFO [DeletionService #3]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,763 INFO [DeletionService #0]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,765 INFO [AsyncDispatcher event handler]
> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
> APPLICATION_STOP for appId application_1391688017341_0001
> 2014-02-06 23:00:54,765 INFO [DeletionService #1]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:54,766 INFO [AsyncDispatcher event handler]
> application.Application (ApplicationImpl.java:handle(451)) - Application
> application_1391688017341_0001 transitioned from
> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
> 2014-02-06 23:00:54,767 INFO [AsyncDispatcher event handler]
> loghandler.NonAggregatingLogHandler
> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion for
> application: application_1391688017341_0001, with delay of 10800 seconds
> 2014-02-06 23:00:54,768 INFO [DeletionService #3]
> nodemanager.DefaultContainerExecutor
> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute path
> :
> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
> 2014-02-06 23:00:58,766 INFO [main] containermanager.ContainerManagerImpl
> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
> waiting for Applications to be Finished. Still alive:
> [application_1391688017341_0001]
> 2014-02-06 23:00:58,768 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55125
> 2014-02-06 23:00:58,771 INFO [IPC Server listener on 55125] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55125
> 2014-02-06 23:00:58,772 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,774 INFO [main] monitor.ContainersMonitorImpl
> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin is
> unavailable on this system.
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> is disabled.
> 2014-02-06 23:01:08,784 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55126
> 2014-02-06 23:01:08,785 INFO [IPC Server listener on 55126] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55126
> 2014-02-06 23:01:08,785 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,785 INFO [Public Localizer]
> localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:run(728)) - Public cache exiting
> 2014-02-06 23:01:08,787 INFO [main] mortbay.log (Slf4jLog.java:info(67))
> - Stopped SelectChannelConnector@Christophers-MacBook-Pro.local:0
> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
> sleep interrupted
> 2014-02-06 23:01:08,891 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
> system...
> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
> stopped.
> 2014-02-06 23:01:08,895 INFO [main] impl.MetricsSystemImpl
> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
> shutdown complete.
> 2014-02-06 23:01:08,896 WARN [ApplicationMaster Launcher]
> amlauncher.ApplicationMasterLauncher
> (ApplicationMasterLauncher.java:run(98)) -
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
> interrupted. Returning.
> 2014-02-06 23:01:08,897 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55122
> 2014-02-06 23:01:08,898 INFO [IPC Server listener on 55122] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55122
> 2014-02-06 23:01:08,898 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55121
> 2014-02-06 23:01:08,898 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,902 INFO [IPC Server listener on 55121] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55121
> 2014-02-06 23:01:08,908 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,902 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55120
> 2014-02-06 23:01:08,912 INFO [IPC Server listener on 55120] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55120
> 2014-02-06 23:01:08,912 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,912 INFO [main] ipc.Server (Server.java:stop(2437)) -
> Stopping server on 55119
> 2014-02-06 23:01:08,917 INFO [IPC Server listener on 55119] ipc.Server
> (Server.java:run(720)) - Stopping IPC Server listener on 55119
> 2014-02-06 23:01:08,917 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> NMLivelinessMonitor thread interrupted
> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
> Returning, interrupted : java.lang.InterruptedException
> 2014-02-06 23:01:08,917 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(866)) - Stopping IPC Server Responder
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> AMLivelinessMonitor thread interrupted
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
> thread interrupted
> 2014-02-06 23:01:08,918 INFO [Ping Checker]
> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127)) -
> AMLivelinessMonitor thread interrupted
>
> Has anyone seen this error before when building hadoop, and does anyone
> have any ideas?
>
> Regards
>
> Chris
>