You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by lk_hadoop <lk...@163.com> on 2019/05/17 03:18:53 UTC
Why yarn job was killed
I can't find error message from logs. I want to know the reason,full log please see the attachments.
2019-05-17
lk_hadoop
发件人："lk_hadoop"<lk...@163.com>
发送时间：2019-05-17 11:02
主题：Kylin Hive Column Cardinality Job killed by user.
收件人："user"<us...@kylin.apache.org>
抄送：
hi,all:
I'm trying to load a hive table and calculate it's Column Cardinality ，but I found the MR job was killed by yarn :
Message from yarn web :
Aggregate Resource Allocation:731002 MB-seconds, 190 vcore-seconds
Logs about the ApplicationMaster syslog :
2019-05-17 10:21:43,167 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1533044360609_6754589_000001
2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1a75e76a)
2019-05-17 10:21:43,874 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-05-17 10:21:43,876 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2019-05-17 10:21:45,201 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-05-17 10:21:45,256 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-05-17 10:21:45,647 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-05-17 10:21:45,648 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-05-17 10:21:45,651 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-05-17 10:21:45,657 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-05-17 10:21:45,658 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-05-17 10:21:45,659 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-05-17 10:21:45,736 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,811 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,874 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,914 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2019-05-17 10:21:45,973 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2019-05-17 10:21:46,392 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2019-05-17 10:21:46,569 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1533044360609_6754589 to jobTokenSecretManager
2019-05-17 10:21:47,160 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1533044360609_6754589 because: not enabled; too many maps; too much input; too much RAM;
2019-05-17 10:21:47,340 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1533044360609_6754589 = 32845338598. Number of splits = 14000
2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1533044360609_6754589 = 1
2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from NEW to INITED
2019-05-17 10:21:47,384 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1533044360609_6754589.
2019-05-17 10:21:47,495 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2019-05-17 10:21:47,530 INFO [Socket Reader #1 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 35208
2019-05-17 10:21:47,540 INFO [Socket Reader #2 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 35208
2019-05-17 10:21:47,555 INFO [Socket Reader #3 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 35208
2019-05-17 10:21:47,592 INFO [Socket Reader #4 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 35208
2019-05-17 10:21:47,628 INFO [Socket Reader #5 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 35208
2019-05-17 10:21:47,709 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2019-05-17 10:21:47,841 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-05-17 10:21:47,856 INFO [IPC Server listener on 35208] org.apache.hadoop.ipc.Server: IPC Server listener on 35208: starting
2019-05-17 10:21:47,875 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at bdp-hdfs-10-10-8-33-39/10.8.33.39:35208
2019-05-17 10:21:48,091 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2019-05-17 10:21:48,102 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2019-05-17 10:21:48,124 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2019-05-17 10:21:48,142 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2019-05-17 10:21:48,249 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2019-05-17 10:21:48,250 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2019-05-17 10:21:48,285 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 35859
2019-05-17 10:21:48,285 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2019-05-17 10:21:48,377 INFO [main] org.mortbay.log: Extract jar:file:/data/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/hadoop-yarn-common-2.6.0-cdh5.14.0.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_35859_mapreduce____fnjnpb/webapp
2019-05-17 10:21:49,050 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:35859
2019-05-17 10:21:49,050 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 35859
2019-05-17 10:21:49,604 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2019-05-17 10:21:49,639 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2019-05-17 10:21:49,653 INFO [Socket Reader #1 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 40180
2019-05-17 10:21:49,666 INFO [Socket Reader #3 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 40180
2019-05-17 10:21:49,667 INFO [Socket Reader #4 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 40180
2019-05-17 10:21:49,667 INFO [Socket Reader #5 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 40180
2019-05-17 10:21:49,702 INFO [Socket Reader #2 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 40180
2019-05-17 10:21:49,773 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-05-17 10:21:49,793 INFO [IPC Server listener on 40180] org.apache.hadoop.ipc.Server: IPC Server listener on 40180: starting
2019-05-17 10:21:50,064 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2019-05-17 10:21:50,241 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:20480, vCores:42>
2019-05-17 10:21:50,242 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.BASE_DATA
2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2019-05-17 10:21:50,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from INITED to SETUP
2019-05-17 10:21:50,300 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2019-05-17 10:21:50,657 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000
2019-05-17 10:21:50,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 2000
2019-05-17 10:21:50,667 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 3000
2019-05-17 10:21:50,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 4000
2019-05-17 10:21:50,670 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 5000
2019-05-17 10:21:50,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 6000
2019-05-17 10:21:50,672 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 7000
2019-05-17 10:21:50,673 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 8000
2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 9000
2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 10000
2019-05-17 10:21:50,675 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 11000
2019-05-17 10:21:50,676 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 12000
2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 13000
2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 14000
....
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013998 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 28000
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013999 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_r_000000 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,460 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED....2019-05-17 10:21:54,532 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2019-05-17 10:21:54,544 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:4096, vCores:1>
2019-05-17 10:21:54,547 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 1000
2019-05-17 10:21:54,551 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 2000
2019-05-17 10:21:54,555 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 3000
2019-05-17 10:21:54,559 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 4000
2019-05-17 10:21:54,584 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 5000
2019-05-17 10:21:54,587 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 6000
2019-05-17 10:21:54,590 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 7000
2019-05-17 10:21:54,648 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 8000
2019-05-17 10:21:54,651 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 9000
2019-05-17 10:21:54,653 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 10000
2019-05-17 10:21:54,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 11000
2019-05-17 10:21:54,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 12000
2019-05-17 10:21:54,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 13000
2019-05-17 10:21:54,903 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:5120, vCores:1>.... I didn't find error log, full syslog please see attachments.
2019-05-17
lk_hadoop
Re: Re: Re: Why yarn job was killed
Posted by lk_hadoop <lk...@163.com>.
sorry. RM log grep by `1533044360609_6754589`. Please see the E-mail attachments.
2019-05-17
lk_hadoop
发件人：Wanqiang Ji <wa...@gmail.com>
发送时间：2019-05-17 15:04
主题：Re: Re: Why yarn job was killed
收件人："lk_hadoop"<lk...@163.com>
抄送："user"<us...@hadoop.apache.org>
It's AM log, we need some RM log grep by `1533044360609_6754589`. Thanks!
On Fri, May 17, 2019 at 2:41 PM lk_hadoop <lk...@163.com> wrote:
I can only find syslog from RM , Please see the E-mail attachments.
2019-05-17
lk_hadoop
发件人：Wanqiang Ji <wa...@gmail.com>
发送时间：2019-05-17 14:28
主题：Re: Why yarn job was killed
收件人："lk_hadoop"<lk...@163.com>
抄送："user"<us...@hadoop.apache.org>
I think you'd better give us some logs about '1533044360609_6754589' in RM.
On Fri, May 17, 2019 at 11:25 AM lk_hadoop <lk...@163.com> wrote:
I can't find error message from logs. I want to know the reason,full log please see the attachments.
2019-05-17
lk_hadoop
发件人："lk_hadoop"<lk...@163.com>
发送时间：2019-05-17 11:02
主题：Kylin Hive Column Cardinality Job killed by user.
收件人："user"<us...@kylin.apache.org>
抄送：
hi,all:
I'm trying to load a hive table and calculate it's Column Cardinality ，but I found the MR job was killed by yarn :
Message from yarn web :
Aggregate Resource Allocation:731002 MB-seconds, 190 vcore-seconds
Logs about the ApplicationMaster syslog :
2019-05-17 10:21:43,167 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1533044360609_6754589_000001
2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1a75e76a)
2019-05-17 10:21:43,874 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-05-17 10:21:43,876 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2019-05-17 10:21:45,201 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-05-17 10:21:45,256 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-05-17 10:21:45,647 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-05-17 10:21:45,648 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-05-17 10:21:45,651 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-05-17 10:21:45,657 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-05-17 10:21:45,658 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-05-17 10:21:45,659 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-05-17 10:21:45,736 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,811 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,874 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,914 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2019-05-17 10:21:45,973 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2019-05-17 10:21:46,392 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2019-05-17 10:21:46,569 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1533044360609_6754589 to jobTokenSecretManager
2019-05-17 10:21:47,160 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1533044360609_6754589 because: not enabled; too many maps; too much input; too much RAM;
2019-05-17 10:21:47,340 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1533044360609_6754589 = 32845338598. Number of splits = 14000
2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1533044360609_6754589 = 1
2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from NEW to INITED
2019-05-17 10:21:47,384 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1533044360609_6754589.
2019-05-17 10:21:47,495 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2019-05-17 10:21:47,530 INFO [Socket Reader #1 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 35208
2019-05-17 10:21:47,540 INFO [Socket Reader #2 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 35208
2019-05-17 10:21:47,555 INFO [Socket Reader #3 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 35208
2019-05-17 10:21:47,592 INFO [Socket Reader #4 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 35208
2019-05-17 10:21:47,628 INFO [Socket Reader #5 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 35208
2019-05-17 10:21:47,709 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2019-05-17 10:21:47,841 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-05-17 10:21:47,856 INFO [IPC Server listener on 35208] org.apache.hadoop.ipc.Server: IPC Server listener on 35208: starting
2019-05-17 10:21:47,875 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at bdp-hdfs-10-10-8-33-39/10.8.33.39:35208
2019-05-17 10:21:48,091 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2019-05-17 10:21:48,102 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2019-05-17 10:21:48,124 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2019-05-17 10:21:48,142 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2019-05-17 10:21:48,249 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2019-05-17 10:21:48,250 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2019-05-17 10:21:48,285 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 35859
2019-05-17 10:21:48,285 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2019-05-17 10:21:48,377 INFO [main] org.mortbay.log: Extract jar:file:/data/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/hadoop-yarn-common-2.6.0-cdh5.14.0.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_35859_mapreduce____fnjnpb/webapp
2019-05-17 10:21:49,050 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:35859
2019-05-17 10:21:49,050 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 35859
2019-05-17 10:21:49,604 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2019-05-17 10:21:49,639 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2019-05-17 10:21:49,653 INFO [Socket Reader #1 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 40180
2019-05-17 10:21:49,666 INFO [Socket Reader #3 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 40180
2019-05-17 10:21:49,667 INFO [Socket Reader #4 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 40180
2019-05-17 10:21:49,667 INFO [Socket Reader #5 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 40180
2019-05-17 10:21:49,702 INFO [Socket Reader #2 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 40180
2019-05-17 10:21:49,773 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-05-17 10:21:49,793 INFO [IPC Server listener on 40180] org.apache.hadoop.ipc.Server: IPC Server listener on 40180: starting
2019-05-17 10:21:50,064 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2019-05-17 10:21:50,241 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:20480, vCores:42>
2019-05-17 10:21:50,242 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.BASE_DATA
2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2019-05-17 10:21:50,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from INITED to SETUP
2019-05-17 10:21:50,300 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2019-05-17 10:21:50,657 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000
2019-05-17 10:21:50,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 2000
2019-05-17 10:21:50,667 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 3000
2019-05-17 10:21:50,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 4000
2019-05-17 10:21:50,670 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 5000
2019-05-17 10:21:50,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 6000
2019-05-17 10:21:50,672 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 7000
2019-05-17 10:21:50,673 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 8000
2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 9000
2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 10000
2019-05-17 10:21:50,675 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 11000
2019-05-17 10:21:50,676 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 12000
2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 13000
2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 14000
....
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013998 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 28000
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013999 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_r_000000 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,460 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED....2019-05-17 10:21:54,532 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2019-05-17 10:21:54,544 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:4096, vCores:1>
2019-05-17 10:21:54,547 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 1000
2019-05-17 10:21:54,551 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 2000
2019-05-17 10:21:54,555 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 3000
2019-05-17 10:21:54,559 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 4000
2019-05-17 10:21:54,584 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 5000
2019-05-17 10:21:54,587 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 6000
2019-05-17 10:21:54,590 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 7000
2019-05-17 10:21:54,648 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 8000
2019-05-17 10:21:54,651 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 9000
2019-05-17 10:21:54,653 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 10000
2019-05-17 10:21:54,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 11000
2019-05-17 10:21:54,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 12000
2019-05-17 10:21:54,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 13000
2019-05-17 10:21:54,903 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:5120, vCores:1>.... I didn't find error log, full syslog please see attachments.
2019-05-17
lk_hadoop
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Re: Why yarn job was killed
Posted by Wanqiang Ji <wa...@gmail.com>.
It's AM log, we need some RM log grep by `1533044360609_6754589`. Thanks!
On Fri, May 17, 2019 at 2:41 PM lk_hadoop <lk...@163.com> wrote:
> I can only find syslog from RM , Please see the E-mail attachments.
>
> 2019-05-17
> ------------------------------
> lk_hadoop
> ------------------------------
>
> *发件人：*Wanqiang Ji <wa...@gmail.com>
> *发送时间：*2019-05-17 14:28
> *主题：*Re: Why yarn job was killed
> *收件人：*"lk_hadoop"<lk...@163.com>
> *抄送：*"user"<us...@hadoop.apache.org>
>
> I think you'd better give us some logs about '1533044360609_6754589' in
> RM.
>
> On Fri, May 17, 2019 at 11:25 AM lk_hadoop <lk...@163.com> wrote:
>
>> I can't find error message from logs. I want to know the reason,full log
>> please see the attachments.
>>
>> 2019-05-17
>> ------------------------------
>> lk_hadoop
>> ------------------------------
>> *发件人：*"lk_hadoop"<lk...@163.com>
>> *发送时间：*2019-05-17 11:02
>> *主题：*Kylin Hive Column Cardinality Job killed by user.
>> *收件人：*"user"<us...@kylin.apache.org>
>> *抄送：*
>>
>> hi,all:
>> I'm trying to load a hive table and calculate it's Column Cardinality
>> ，but I found the MR job was killed by yarn :
>>
>> Message from yarn web :
>>
>> Aggregate Resource Allocation: 731002 MB-seconds, 190 vcore-seconds
>>
>> Logs about the ApplicationMaster syslog :
>>
>> 2019-05-17 10:21:43,167 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1533044360609_6754589_000001
>> 2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
>> 2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1a75e76a)
>> 2019-05-17 10:21:43,874 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
>> 2019-05-17 10:21:43,876 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
>> 2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
>> 2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
>> 2019-05-17 10:21:45,201 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>> 2019-05-17 10:21:45,256 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
>> 2019-05-17 10:21:45,647 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
>> 2019-05-17 10:21:45,648 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
>> 2019-05-17 10:21:45,651 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
>> 2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
>> 2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
>> 2019-05-17 10:21:45,657 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
>> 2019-05-17 10:21:45,658 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
>> 2019-05-17 10:21:45,659 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
>> 2019-05-17 10:21:45,736 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
>> 2019-05-17 10:21:45,811 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
>> 2019-05-17 10:21:45,874 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
>> 2019-05-17 10:21:45,914 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
>> 2019-05-17 10:21:45,973 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
>> 2019-05-17 10:21:46,392 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
>> 2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
>> 2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
>> 2019-05-17 10:21:46,569 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1533044360609_6754589 to jobTokenSecretManager
>> 2019-05-17 10:21:47,160 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1533044360609_6754589 because: not enabled; too many maps; too much input; too much RAM;
>> 2019-05-17 10:21:47,340 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1533044360609_6754589 = 32845338598. Number of splits = 14000
>> 2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1533044360609_6754589 = 1
>> 2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from NEW to INITED
>> 2019-05-17 10:21:47,384 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1533044360609_6754589.
>> 2019-05-17 10:21:47,495 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
>> 2019-05-17 10:21:47,530 INFO [Socket Reader #1 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 35208
>> 2019-05-17 10:21:47,540 INFO [Socket Reader #2 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 35208
>> 2019-05-17 10:21:47,555 INFO [Socket Reader #3 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 35208
>> 2019-05-17 10:21:47,592 INFO [Socket Reader #4 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 35208
>> 2019-05-17 10:21:47,628 INFO [Socket Reader #5 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 35208
>> 2019-05-17 10:21:47,709 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
>> 2019-05-17 10:21:47,841 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>> 2019-05-17 10:21:47,856 INFO [IPC Server listener on 35208] org.apache.hadoop.ipc.Server: IPC Server listener on 35208: starting
>> 2019-05-17 10:21:47,875 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at bdp-hdfs-10-10-8-33-39/10.8.33.39:35208
>> 2019-05-17 10:21:48,091 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>> 2019-05-17 10:21:48,102 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
>> 2019-05-17 10:21:48,124 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
>> 2019-05-17 10:21:48,142 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
>> 2019-05-17 10:21:48,249 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
>> 2019-05-17 10:21:48,250 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
>> 2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
>> 2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
>> 2019-05-17 10:21:48,285 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 35859
>> 2019-05-17 10:21:48,285 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
>> 2019-05-17 10:21:48,377 INFO [main] org.mortbay.log: Extract jar:file:/data/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/hadoop-yarn-common-2.6.0-cdh5.14.0.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_35859_mapreduce____fnjnpb/webapp
>> 2019-05-17 10:21:49,050 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:35859
>> 2019-05-17 10:21:49,050 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 35859
>> 2019-05-17 10:21:49,604 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
>> 2019-05-17 10:21:49,639 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
>> 2019-05-17 10:21:49,653 INFO [Socket Reader #1 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 40180
>> 2019-05-17 10:21:49,666 INFO [Socket Reader #3 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 40180
>> 2019-05-17 10:21:49,667 INFO [Socket Reader #4 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 40180
>> 2019-05-17 10:21:49,667 INFO [Socket Reader #5 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 40180
>> 2019-05-17 10:21:49,702 INFO [Socket Reader #2 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 40180
>> 2019-05-17 10:21:49,773 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
>> 2019-05-17 10:21:49,793 INFO [IPC Server listener on 40180] org.apache.hadoop.ipc.Server: IPC Server listener on 40180: starting
>> 2019-05-17 10:21:50,064 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
>> 2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
>> 2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
>> 2019-05-17 10:21:50,241 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:20480, vCores:42>
>> 2019-05-17 10:21:50,242 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.BASE_DATA
>> 2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
>> 2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
>> 2019-05-17 10:21:50,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from INITED to SETUP
>> 2019-05-17 10:21:50,300 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
>> 2019-05-17 10:21:50,657 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000
>> 2019-05-17 10:21:50,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 2000
>> 2019-05-17 10:21:50,667 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 3000
>> 2019-05-17 10:21:50,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 4000
>> 2019-05-17 10:21:50,670 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 5000
>> 2019-05-17 10:21:50,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 6000
>> 2019-05-17 10:21:50,672 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 7000
>> 2019-05-17 10:21:50,673 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 8000
>> 2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 9000
>> 2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 10000
>> 2019-05-17 10:21:50,675 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 11000
>> 2019-05-17 10:21:50,676 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 12000
>> 2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 13000
>> 2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 14000
>>
>> ....
>>
>> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013998 Task Transitioned from NEW to SCHEDULED
>> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 28000
>> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013999 Task Transitioned from NEW to SCHEDULED
>> 2019-05-17 10:21:53,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_r_000000 Task Transitioned from NEW to SCHEDULED
>> 2019-05-17 10:21:53,460 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
>>
>> ....
>>
>> 2019-05-17 10:21:54,532 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
>> 2019-05-17 10:21:54,544 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:4096, vCores:1>
>> 2019-05-17 10:21:54,547 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 1000
>> 2019-05-17 10:21:54,551 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 2000
>> 2019-05-17 10:21:54,555 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 3000
>> 2019-05-17 10:21:54,559 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 4000
>> 2019-05-17 10:21:54,584 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 5000
>> 2019-05-17 10:21:54,587 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 6000
>> 2019-05-17 10:21:54,590 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 7000
>> 2019-05-17 10:21:54,648 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 8000
>> 2019-05-17 10:21:54,651 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 9000
>> 2019-05-17 10:21:54,653 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 10000
>> 2019-05-17 10:21:54,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 11000
>> 2019-05-17 10:21:54,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 12000
>> 2019-05-17 10:21:54,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 13000
>> 2019-05-17 10:21:54,903 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:5120, vCores:1>
>>
>> ....
>>
>>
>>
>> I didn't find error log, full syslog please see attachments.
>>
>>
>> 2019-05-17
>> ------------------------------
>> lk_hadoop
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>
>
Re: Re: Why yarn job was killed
Posted by lk_hadoop <lk...@163.com>.
I can only find syslog from RM , Please see the E-mail attachments.
2019-05-17
lk_hadoop
发件人：Wanqiang Ji <wa...@gmail.com>
发送时间：2019-05-17 14:28
主题：Re: Why yarn job was killed
收件人："lk_hadoop"<lk...@163.com>
抄送："user"<us...@hadoop.apache.org>
I think you'd better give us some logs about '1533044360609_6754589' in RM.
On Fri, May 17, 2019 at 11:25 AM lk_hadoop <lk...@163.com> wrote:
I can't find error message from logs. I want to know the reason,full log please see the attachments.
2019-05-17
lk_hadoop
发件人："lk_hadoop"<lk...@163.com>
发送时间：2019-05-17 11:02
主题：Kylin Hive Column Cardinality Job killed by user.
收件人："user"<us...@kylin.apache.org>
抄送：
hi,all:
I'm trying to load a hive table and calculate it's Column Cardinality ，but I found the MR job was killed by yarn :
Message from yarn web :
Aggregate Resource Allocation:731002 MB-seconds, 190 vcore-seconds
Logs about the ApplicationMaster syslog :
2019-05-17 10:21:43,167 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1533044360609_6754589_000001
2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1a75e76a)
2019-05-17 10:21:43,874 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2019-05-17 10:21:43,876 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2019-05-17 10:21:45,201 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-05-17 10:21:45,256 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-05-17 10:21:45,647 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2019-05-17 10:21:45,648 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2019-05-17 10:21:45,651 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2019-05-17 10:21:45,657 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2019-05-17 10:21:45,658 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2019-05-17 10:21:45,659 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2019-05-17 10:21:45,736 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,811 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,874 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
2019-05-17 10:21:45,914 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2019-05-17 10:21:45,973 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2019-05-17 10:21:46,392 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2019-05-17 10:21:46,569 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1533044360609_6754589 to jobTokenSecretManager
2019-05-17 10:21:47,160 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1533044360609_6754589 because: not enabled; too many maps; too much input; too much RAM;
2019-05-17 10:21:47,340 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1533044360609_6754589 = 32845338598. Number of splits = 14000
2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1533044360609_6754589 = 1
2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from NEW to INITED
2019-05-17 10:21:47,384 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1533044360609_6754589.
2019-05-17 10:21:47,495 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2019-05-17 10:21:47,530 INFO [Socket Reader #1 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 35208
2019-05-17 10:21:47,540 INFO [Socket Reader #2 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 35208
2019-05-17 10:21:47,555 INFO [Socket Reader #3 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 35208
2019-05-17 10:21:47,592 INFO [Socket Reader #4 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 35208
2019-05-17 10:21:47,628 INFO [Socket Reader #5 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 35208
2019-05-17 10:21:47,709 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2019-05-17 10:21:47,841 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-05-17 10:21:47,856 INFO [IPC Server listener on 35208] org.apache.hadoop.ipc.Server: IPC Server listener on 35208: starting
2019-05-17 10:21:47,875 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at bdp-hdfs-10-10-8-33-39/10.8.33.39:35208
2019-05-17 10:21:48,091 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2019-05-17 10:21:48,102 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2019-05-17 10:21:48,124 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2019-05-17 10:21:48,142 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2019-05-17 10:21:48,249 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2019-05-17 10:21:48,250 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2019-05-17 10:21:48,285 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 35859
2019-05-17 10:21:48,285 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2019-05-17 10:21:48,377 INFO [main] org.mortbay.log: Extract jar:file:/data/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/hadoop-yarn-common-2.6.0-cdh5.14.0.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_35859_mapreduce____fnjnpb/webapp
2019-05-17 10:21:49,050 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:35859
2019-05-17 10:21:49,050 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 35859
2019-05-17 10:21:49,604 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2019-05-17 10:21:49,639 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2019-05-17 10:21:49,653 INFO [Socket Reader #1 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 40180
2019-05-17 10:21:49,666 INFO [Socket Reader #3 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 40180
2019-05-17 10:21:49,667 INFO [Socket Reader #4 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 40180
2019-05-17 10:21:49,667 INFO [Socket Reader #5 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 40180
2019-05-17 10:21:49,702 INFO [Socket Reader #2 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 40180
2019-05-17 10:21:49,773 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2019-05-17 10:21:49,793 INFO [IPC Server listener on 40180] org.apache.hadoop.ipc.Server: IPC Server listener on 40180: starting
2019-05-17 10:21:50,064 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2019-05-17 10:21:50,241 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:20480, vCores:42>
2019-05-17 10:21:50,242 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.BASE_DATA
2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2019-05-17 10:21:50,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from INITED to SETUP
2019-05-17 10:21:50,300 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2019-05-17 10:21:50,657 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000
2019-05-17 10:21:50,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 2000
2019-05-17 10:21:50,667 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 3000
2019-05-17 10:21:50,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 4000
2019-05-17 10:21:50,670 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 5000
2019-05-17 10:21:50,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 6000
2019-05-17 10:21:50,672 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 7000
2019-05-17 10:21:50,673 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 8000
2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 9000
2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 10000
2019-05-17 10:21:50,675 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 11000
2019-05-17 10:21:50,676 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 12000
2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 13000
2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 14000
....
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013998 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 28000
2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013999 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_r_000000 Task Transitioned from NEW to SCHEDULED
2019-05-17 10:21:53,460 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED....2019-05-17 10:21:54,532 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2019-05-17 10:21:54,544 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:4096, vCores:1>
2019-05-17 10:21:54,547 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 1000
2019-05-17 10:21:54,551 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 2000
2019-05-17 10:21:54,555 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 3000
2019-05-17 10:21:54,559 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 4000
2019-05-17 10:21:54,584 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 5000
2019-05-17 10:21:54,587 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 6000
2019-05-17 10:21:54,590 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 7000
2019-05-17 10:21:54,648 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 8000
2019-05-17 10:21:54,651 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 9000
2019-05-17 10:21:54,653 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 10000
2019-05-17 10:21:54,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 11000
2019-05-17 10:21:54,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 12000
2019-05-17 10:21:54,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 13000
2019-05-17 10:21:54,903 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:5120, vCores:1>.... I didn't find error log, full syslog please see attachments.
2019-05-17
lk_hadoop
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Why yarn job was killed
Posted by Wanqiang Ji <wa...@gmail.com>.
I think you'd better give us some logs about '1533044360609_6754589' in RM.
On Fri, May 17, 2019 at 11:25 AM lk_hadoop <lk...@163.com> wrote:
> I can't find error message from logs. I want to know the reason,full log
> please see the attachments.
>
> 2019-05-17
> ------------------------------
> lk_hadoop
> ------------------------------
> *发件人：*"lk_hadoop"<lk...@163.com>
> *发送时间：*2019-05-17 11:02
> *主题：*Kylin Hive Column Cardinality Job killed by user.
> *收件人：*"user"<us...@kylin.apache.org>
> *抄送：*
>
> hi,all:
> I'm trying to load a hive table and calculate it's Column Cardinality
> ，but I found the MR job was killed by yarn :
>
> Message from yarn web :
>
> Aggregate Resource Allocation: 731002 MB-seconds, 190 vcore-seconds
>
> Logs about the ApplicationMaster syslog :
>
> 2019-05-17 10:21:43,167 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1533044360609_6754589_000001
> 2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2019-05-17 10:21:43,614 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1a75e76a)
> 2019-05-17 10:21:43,874 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
> 2019-05-17 10:21:43,876 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
> 2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
> 2019-05-17 10:21:43,994 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
> 2019-05-17 10:21:45,201 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2019-05-17 10:21:45,256 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 2019-05-17 10:21:45,647 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
> 2019-05-17 10:21:45,648 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
> 2019-05-17 10:21:45,651 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
> 2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
> 2019-05-17 10:21:45,653 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
> 2019-05-17 10:21:45,657 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
> 2019-05-17 10:21:45,658 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
> 2019-05-17 10:21:45,659 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
> 2019-05-17 10:21:45,736 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
> 2019-05-17 10:21:45,811 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
> 2019-05-17 10:21:45,874 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
> 2019-05-17 10:21:45,914 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
> 2019-05-17 10:21:45,973 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
> 2019-05-17 10:21:46,392 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2019-05-17 10:21:46,543 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
> 2019-05-17 10:21:46,569 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1533044360609_6754589 to jobTokenSecretManager
> 2019-05-17 10:21:47,160 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1533044360609_6754589 because: not enabled; too many maps; too much input; too much RAM;
> 2019-05-17 10:21:47,340 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1533044360609_6754589 = 32845338598. Number of splits = 14000
> 2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1533044360609_6754589 = 1
> 2019-05-17 10:21:47,383 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from NEW to INITED
> 2019-05-17 10:21:47,384 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1533044360609_6754589.
> 2019-05-17 10:21:47,495 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
> 2019-05-17 10:21:47,530 INFO [Socket Reader #1 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 35208
> 2019-05-17 10:21:47,540 INFO [Socket Reader #2 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 35208
> 2019-05-17 10:21:47,555 INFO [Socket Reader #3 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 35208
> 2019-05-17 10:21:47,592 INFO [Socket Reader #4 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 35208
> 2019-05-17 10:21:47,628 INFO [Socket Reader #5 for port 35208] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 35208
> 2019-05-17 10:21:47,709 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
> 2019-05-17 10:21:47,841 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2019-05-17 10:21:47,856 INFO [IPC Server listener on 35208] org.apache.hadoop.ipc.Server: IPC Server listener on 35208: starting
> 2019-05-17 10:21:47,875 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at bdp-hdfs-10-10-8-33-39/10.8.33.39:35208
> 2019-05-17 10:21:48,091 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2019-05-17 10:21:48,102 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
> 2019-05-17 10:21:48,124 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
> 2019-05-17 10:21:48,142 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2019-05-17 10:21:48,249 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
> 2019-05-17 10:21:48,250 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
> 2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
> 2019-05-17 10:21:48,263 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
> 2019-05-17 10:21:48,285 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 35859
> 2019-05-17 10:21:48,285 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
> 2019-05-17 10:21:48,377 INFO [main] org.mortbay.log: Extract jar:file:/data/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/hadoop-yarn-common-2.6.0-cdh5.14.0.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_35859_mapreduce____fnjnpb/webapp
> 2019-05-17 10:21:49,050 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:35859
> 2019-05-17 10:21:49,050 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 35859
> 2019-05-17 10:21:49,604 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
> 2019-05-17 10:21:49,639 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
> 2019-05-17 10:21:49,653 INFO [Socket Reader #1 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 40180
> 2019-05-17 10:21:49,666 INFO [Socket Reader #3 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #3 for port 40180
> 2019-05-17 10:21:49,667 INFO [Socket Reader #4 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #4 for port 40180
> 2019-05-17 10:21:49,667 INFO [Socket Reader #5 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #5 for port 40180
> 2019-05-17 10:21:49,702 INFO [Socket Reader #2 for port 40180] org.apache.hadoop.ipc.Server: Starting Socket Reader #2 for port 40180
> 2019-05-17 10:21:49,773 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2019-05-17 10:21:49,793 INFO [IPC Server listener on 40180] org.apache.hadoop.ipc.Server: IPC Server listener on 40180: starting
> 2019-05-17 10:21:50,064 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
> 2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
> 2019-05-17 10:21:50,065 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
> 2019-05-17 10:21:50,241 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:20480, vCores:42>
> 2019-05-17 10:21:50,242 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.BASE_DATA
> 2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
> 2019-05-17 10:21:50,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
> 2019-05-17 10:21:50,297 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1533044360609_6754589Job Transitioned from INITED to SETUP
> 2019-05-17 10:21:50,300 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
> 2019-05-17 10:21:50,657 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 1000
> 2019-05-17 10:21:50,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 2000
> 2019-05-17 10:21:50,667 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 3000
> 2019-05-17 10:21:50,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 4000
> 2019-05-17 10:21:50,670 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 5000
> 2019-05-17 10:21:50,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 6000
> 2019-05-17 10:21:50,672 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 7000
> 2019-05-17 10:21:50,673 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 8000
> 2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 9000
> 2019-05-17 10:21:50,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 10000
> 2019-05-17 10:21:50,675 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 11000
> 2019-05-17 10:21:50,676 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 12000
> 2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 13000
> 2019-05-17 10:21:50,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 14000
>
> ....
>
> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013998 Task Transitioned from NEW to SCHEDULED
> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 28000
> 2019-05-17 10:21:53,440 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_m_013999 Task Transitioned from NEW to SCHEDULED
> 2019-05-17 10:21:53,441 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1533044360609_6754589_r_000000 Task Transitioned from NEW to SCHEDULED
> 2019-05-17 10:21:53,460 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
>
> ....
>
> 2019-05-17 10:21:54,532 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1533044360609_6754589_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
> 2019-05-17 10:21:54,544 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:4096, vCores:1>
> 2019-05-17 10:21:54,547 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 1000
> 2019-05-17 10:21:54,551 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 2000
> 2019-05-17 10:21:54,555 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 3000
> 2019-05-17 10:21:54,559 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 4000
> 2019-05-17 10:21:54,584 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 5000
> 2019-05-17 10:21:54,587 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 6000
> 2019-05-17 10:21:54,590 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 7000
> 2019-05-17 10:21:54,648 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 8000
> 2019-05-17 10:21:54,651 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 9000
> 2019-05-17 10:21:54,653 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 10000
> 2019-05-17 10:21:54,666 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 11000
> 2019-05-17 10:21:54,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 12000
> 2019-05-17 10:21:54,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Size of event-queue in RMContainerAllocator is 13000
> 2019-05-17 10:21:54,903 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:5120, vCores:1>
>
> ....
>
>
>
> I didn't find error log, full syslog please see attachments.
>
>
> 2019-05-17
> ------------------------------
> lk_hadoop
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org