You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Hitesh Shah (JIRA)" <ji...@apache.org> on 2015/04/02 03:08:53 UTC
[jira] [Updated] (TEZ-2263) AM crashes on DAG completion if counter
limits are exceeded.
[ https://issues.apache.org/jira/browse/TEZ-2263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated TEZ-2263:
-----------------------------
Fix Version/s: (was: 0.5.0)
> AM crashes on DAG completion if counter limits are exceeded.
> -------------------------------------------------------------
>
> Key: TEZ-2263
> URL: https://issues.apache.org/jira/browse/TEZ-2263
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Mostafa Mokhtar
>
> Commit fails then Tez tries to recover which fails again.
> {code}
> 5499174247-2015-04-01 16:23:20,600 INFO [main] app.RecoveryParser: Found summary file in attempt directory, summaryFile=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1/summary, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1
> 5499174696-2015-04-01 16:23:20,600 INFO [main] app.RecoveryParser: Using hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1 for recovering data from previous attempt
> 5499174963-2015-04-01 16:23:20,690 INFO [main] app.RecoveryParser: Parsing summary file, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1/summary, len=4024, lastModTime=1427919788998
> 5499175254-2015-04-01 16:23:20,786 INFO [main] app.RecoveryParser: Reached end of summary stream
> 5499175340-2015-04-01 16:23:21,087 INFO [main] app.RecoveryParser: Checking if DAG is in recoverable state, dagId=dag_1426707664723_1086_1
> 5499175468-2015-04-01 16:23:21,088 WARN [main] app.RecoveryParser: Found last inProgress DAG but not recoverable: dagId=dag_1426707664723_1086_1, dagCompleted=false
> 5499175622-2015-04-01 16:23:21,088 INFO [main] app.RecoveryParser: Trying to recover dag from recovery file, dagId=dag_1426707664723_1086_1, dataDir=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1, intoCurrentDir=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2
> 5499176102-2015-04-01 16:23:21,091 INFO [main] app.RecoveryParser: Copying DAG data into Current Attempt directory, filePath=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2/dag_1426707664723_1086_1.recovery
> 5499176413-2015-04-01 16:23:21,211 INFO [main] app.RecoveryParser: Recovering from event, eventType=DAG_SUBMITTED, event=dagID=dag_1426707664723_1086_1, submitTime=1427917169723
> 5499176580-2015-04-01 16:23:21,309 INFO [main] app.DAGAppMaster: Generating DAG graphviz file, dagId=dag_1426707664723_1086_1, filePath=/grid/0/cluster/yarn/log/application_1426707664723_1086/container_1426707664723_1086_02_000001/dag_1426707664723_1086_1.dot
> 5499176829-2015-04-01 16:23:21,347 INFO [main] app.DAGAppMaster: Writing DAG plan to: /grid/0/cluster/yarn/log/application_1426707664723_1086/container_1426707664723_1086_02_000001/dag_1426707664723_1086_1-tez-dag.pb.txt
> 5499177039-2015-04-01 16:23:22,576 INFO [main] app.RecoveryParser: Finished copying data from previous attempt into current attempt
> 5499177160-2015-04-01 16:23:22,576 INFO [main] app.RecoveryParser: Trying to create data recovered flag file, filePath=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2/dataRecovered
> 5499177445-2015-04-01 16:23:22,601 INFO [main] app.DAGAppMaster: In Session mode. Waiting for DAG over RPC
> 5499177541-2015-04-01 16:23:22,601 INFO [main] app.DAGAppMaster: Found previous DAG in completed or non-recoverable state, dagId=dag_1426707664723_1086_1, isCompleted=false, isNonRecoverable=true, state=null, failureReason=DAG Commit was in progress, not recoverable, dagId=dag_1426707664723_1086_1
> 5499177829-2015-04-01 16:23:22,601 INFO [main] common.TezUtilsInternal: Redirecting log file based on addend: dag_1426707664723_1086_1
> 5499177953-
> 5499177954-LogType:syslog_dag_1426707664723_1086_1
> 5499177994-Log Upload Time:1-Apr-2015 16:24:30
> 5499178030-LogLength:521
> 5499178044-Log Contents:
> 5499178058-2015-04-01 16:23:22,604 INFO [main] impl.DAGImpl: Recovered DAG: dag_1426707664723_1086_1 finished with state: FAILED
> 5499178176-2015-04-01 16:23:22,604 INFO [main] impl.DAGImpl: dag_1426707664723_1086_1 transitioned from NEW to FAILED
> 5499178283-2015-04-01 16:23:22,604 INFO [AsyncDispatcher event handler] app.DAGAppMaster: DAG completed, dagId=dag_1426707664723_1086_1, dagState=FAILED
> 5499178425-2015-04-01 16:23:22,605 INFO [AsyncDispatcher event handler] common.TezUtilsInternal: Redirecting log file based on addend: dag_1426707664723_1086_1_post
> 5499178579-
> 5499178580-LogType:syslog_dag_1426707664723_1086_1_post
> 5499178625-Log Upload Time:1-Apr-2015 16:24:30
> 5499178661-LogLength:4021
> 5499178676-Log Contents:
> 5499178690-2015-04-01 16:23:22,605 INFO [AsyncDispatcher event handler] app.DAGAppMaster: Waiting for next DAG to be submitted.
> 5499178807-2015-04-01 16:24:01,681 INFO [IPC Server handler 0 on 53890] client.DAGClientHandler: Received message to shutdown AM
> 5499178925-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] rm.TaskSchedulerEventHandler: TaskScheduler notified that it should unregister from RM
> 5499179073-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: No current running DAG, shutting down the AM
> 5499179197-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: DAGAppMasterShutdownHandler invoked
> 5499179312-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: Handling DAGAppMaster shutdown
> 5499179422-2015-04-01 16:24:01,683 INFO [AMShutdownThread] app.DAGAppMaster: Sleeping for 5 seconds before shutting down
> 5499179532-2015-04-01 16:24:04,151 INFO [HistoryEventHandlingThread] ats.ATSHistoryLoggingService: Event queue stats, eventsProcessedSinceLastUpdate=2, eventQueueSize=0
> 5499179690-2015-04-01 16:24:06,683 INFO [AMShutdownThread] app.DAGAppMaster: Calling stop for all the services
> 5499179790-2015-04-01 16:24:06,686 INFO [AMShutdownThread] history.HistoryEventHandler: Stopping HistoryEventHandler
> 5499179896-2015-04-01 16:24:06,686 INFO [AMShutdownThread] recovery.RecoveryService: Stopping RecoveryService
> 5499179995-2015-04-01 16:24:06,686 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0
> 5499180114-2015-04-01 16:24:06,686 INFO [RecoveryEventHandlingThread] recovery.RecoveryService: EventQueue take interrupted. Returning
> 5499180238-2015-04-01 16:24:06,692 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted
> 5499180367-2015-04-01 16:24:06,697 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=0, successfulDAGs=0, failedDAGs=1, killedDAGs=0
> 5499180589-, trackingURL=
> 5499180604-2015-04-01 16:24:06,713 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
> 5499180730-2015-04-01 16:24:06,819 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Successfully unregistered application from RM
> 5499180853-2015-04-01 16:24:06,821 INFO [AMShutdownThread] ipc.Server: Stopping server on 50998
> 5499180938-2015-04-01 16:24:06,821 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl: Interrupted while waiting for queue
> 5499181060:java.lang.InterruptedException
> 5499181091- at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 5499181228- at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
> 5499181346- at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> 5499181426- at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
> 5499181551-2015-04-01 16:24:06,823 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder
> 5499181645-2015-04-01 16:24:06,822 INFO [IPC Server listener on 50998] ipc.Server: Stopping IPC Server listener on 50998
> 5499181755-2015-04-01 16:24:06,822 INFO [AMShutdownThread] ipc.Server: Stopping server on 53890
> 5499181840-2015-04-01 16:24:06,826 INFO [IPC Server listener on 53890] ipc.Server: Stopping IPC Server listener on 53890
> 5499181950-2015-04-01 16:24:06,826 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder
> 5499182044-2015-04-01 16:24:06,828 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked
> 5499182135-2015-04-01 16:24:06,828 INFO [Thread-1] app.DAGAppMaster: The shutdown handler is still running, waiting for it to complete
> 5499182259-2015-04-01 16:24:06,839 WARN [AMShutdownThread] app.DAGAppMaster: Failed to delete tez scratch data dir, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086
> 5499182521-2015-04-01 16:24:06,839 INFO [AMShutdownThread] app.DAGAppMaster: Exiting DAGAppMaster..GoodBye!
> 5499182618-2015-04-01 16:24:06,839 INFO [Thread-1] app.DAGAppMaster: The shutdown handler has completed
> 5499182711-
> 5499182712-
> 5499182713-
> 5499182714-Container: container_1426707664723_1086_01_000304 on cn110-10.l42scl.hortonworks.com_45454
> 5499182805-============================================================================================
> 5499182898-LogType:stderr
> 5499182913-Log Upload Time:1-Apr-2015 16:24:30
> 5499182949-LogLength:0
> 5499182961-Log Contents:
> 5499182975-
> 5499182976-LogType:stdout
> 5499182991-Log Upload Time:1-Apr-2015 16:24:30
> 5499183027-LogLength:81042
> 5499183043-Log Contents:
> 5499183057-9.076: [GC [PSYoungGen: 415194K->108031K(756736K)] 1463770K->1182454K(7674880K), 0.1355580 secs] [Times: user=0.59 sys=0.13, real=0.13 secs]
> 5499183199-16.885: [GC [PSYoungGen: 446979K->70470K(756736K)] 1521402K->1144901K(7674880K), 0.1141880 secs] [Times: user=0.41 sys=0.08, real=0.11 secs]
> 5499183341-32.610: [GC [PSYoungGen: 401811K->23054K(756736K)] 1476242K->1097493K(7674880K), 0.0198940 secs] [Times: user=0.26 sys=0.00, real=0.02 secs]
> 5499183483-37.397: [GC [PSYoungGen: 354557K->108009K(756736K)] 2477572K->2246623K(7674880K), 0.0591560 secs] [Times: user=0.82 sys=0.06, real=0.06 secs]
> 5499183626-42.607: [GC [PSYoungGen: 434535K->15928K(649216K)] 2573148K->2154549K(7567360K), 0.0639210 secs] [Times: user=0.45 sys=0.03, real=0.06 secs]
> 5499183768-47.641: [GC [PSYoungGen: 553804K->22893K(689152K)] 3741001K->3210098K(7607296K), 0.0727270 secs] [Times: user=0.55 sys=0.06, real=0.07 secs]
> 5499183910-53.107: [GC [PSYoungGen: 343874K->137457K(646656K)] 4579655K->4384105K(7564800K), 0.1012020 secs] [Times: user=0.50 sys=0.14, real=0.10 secs]
> 5499184053-62.586: [GC [PSYoungGen: 484074K->89878K(676864K)] 4730722K->4336778K(7595008K), 0.0985300 secs] [Times: user=0.56 sys=0.08, real=0.10 secs]
> 5499184195-76.005: [GC [PSYoungGen: 449294K->16827K(668672K)] 4696194K->4274117K(7586816K), 0.0315610 secs] [Times: user=0.36 sys=0.03, real=0.03 secs]
> 5499184337-79.777: [GC [PSYoungGen: 211179K->19599K(680448K)] 5517045K->5332144K(7598592K), 0.0304100 secs] [Times: user=0.37 sys=0.02, real=0.03 secs]
> 5499184479-81.315: [GC [PSYoungGen: 150008K->78090K(677888K)] 5462554K->5399597K(7596032K), 0.0293570 secs] [Times: user=0.39 sys=0.02, real=0.03 secs]
> 5499184621-82.455: [GC [PSYoungGen: 237557K->512K(687616K)] 5559064K->5324779K(7605760K), 0.0384990 secs] [Times: user=0.34 sys=0.02, real=0.04 secs]
> 5499184761-84.067: [GC [PSYoungGen: 210827K->14117K(687616K)] 6583671K->6387049K(7605760K), 0.0517180 secs] [Times: user=0.66 sys=0.01, real=0.05 secs]
> 5499184903-88.416: [GC [PSYoungGen: 268920K->15351K(696320K)] 6641851K->6394989K(7614464K), 0.0787950 secs] [Times: user=0.51 sys=0.02, real=0.08 secs]
> 5499185045-101.043: [GC [PSYoungGen: 376721K->448K(691200K)] 6756359K->6387846K(7609344K), 0.0282280 secs] [Times: user=0.44 sys=0.02, real=0.03 secs]
> 5499185186-103.105: [GC [PSYoungGen: 82965K->13643K(705536K)] 6470363K->6401129K(7623680K), 0.1482160 secs] [Times: user=0.53 sys=0.00, real=0.15 secs]
> 5499185328-103.253: [GC [PSYoungGen: 13643K->0K(699392K)] 6401129K->6400834K(7617536K), 0.0805200 secs] [Times: user=0.56 sys=0.02, real=0.08 secs]
> 5499185466-103.334: [Full GC [PSYoungGen: 0K->0K(699392K)] [ParOldGen: 6400834K->26007K(6918144K)] 6400834K->26007K(7617536K) [PSPermGen: 33081K->33057K(66560K)], 0.4079400 secs] [Times: user=1.08 sys=0.17, real=0.41 secs]
> 5499185679-108.478: [GC [PSYoungGen: 279515K->23409K(718848K)] 1354098K->1097992K(7636992K), 0.0143960 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
> 5499185822-121.280: [GC [PSYoungGen: 322641K->279K(709632K)] 1397224K->1082530K(7627776K), 0.0187460 secs] [Times: user=0.10 sys=0.00, real=0.01 secs]
> 5499185963-126.595: [GC [PSYoungGen: 345663K->20073K(731648K)] 2476491K->2150964K(7649792K), 0.0305890 secs] [Times: user=0.19 sys=0.00, real=0.03 secs]
> 5499186106-141.120: [GC [PSYoungGen: 596191K->14241K(723456K)] 3775658K->3200292K(7641600K), 0.0189990 secs] [Times: user=0.27 sys=0.00, real=0.02 secs]
> --
> 18043554745-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 49
> 18043554860-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 44
> 18043554975-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 45
> 18043555090-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 50
> 18043555205-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 25
> 18043555320-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 17
> 18043555435-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 23
> 18043555550-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 24
> 18043555665-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 21
> 18043555780-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 22
> 18043555895-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 38
> 18043556014-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 20
> 18043556129-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 31
> 18043556248-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 37
> 18043556367-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 29
> 18043556482-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 30
> 18043556601-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 27
> 18043556716-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 28
> 18043556831-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 3
> 18043556949-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 26
> 18043557064-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 2
> 18043557182-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 32
> 18043557297-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 36
> 18043557416-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 35
> 18043557535-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 34
> 18043557654-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 33
> 18043557769-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 1
> 18043557883-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 39
> 18043557998-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 5
> 18043558112-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 4
> 18043558226-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 43
> 18043558341-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 53
> 18043558456-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 42
> 18043558571-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 54
> 18043558686-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 41
> 18043558801-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 51
> 18043558916-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 14
> 18043559031-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 7
> 18043559149-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 13
> 18043559264-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 6
> 18043559382-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 16
> 18043559497-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 19
> 18043559612-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 18
> 18043559727-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 15
> 18043559842-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No exclusive output committers for vertex: Reducer 12
> 18043559971-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 10
> 18043560090-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 11
> 18043560209-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 8
> 18043560327-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 9
> 18043560445-2015-04-01 16:23:08,857 FATAL [AsyncDispatcher event handler] event.AsyncDispatcher: Error in dispatcher thread
> 18043560557:org.apache.tez.common.counters.LimitExceededException: Too many counters: 1201 max=1200
> 18043560645- at org.apache.tez.common.counters.Limits.checkCounters(Limits.java:87)
> 18043560717- at org.apache.tez.common.counters.Limits.incrCounters(Limits.java:94)
> 18043560788- at org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:75)
> 18043560885- at org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
> 18043560986- at org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:103)
> 18043561085- at org.apache.tez.common.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:198)
> 18043561188- at org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:363)
> 18043561283- at org.apache.tez.dag.app.dag.impl.DAGImpl.incrTaskCounters(DAGImpl.java:598)
> 18043561362- at org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:588)
> 18043561439- at org.apache.tez.dag.app.dag.impl.DAGImpl.logJobHistoryFinishedEvent(DAGImpl.java:994)
> 18043561528- at org.apache.tez.dag.app.dag.impl.DAGImpl.finished(DAGImpl.java:1135)
> 18043561600- at org.apache.tez.dag.app.dag.impl.DAGImpl.checkDAGForCompletion(DAGImpl.java:1048)
> 18043561685- at org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1708)
> 18043561785- at org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1665)
> 18043561885- at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> 18043562001- at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> 18043562097- at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> 18043562190- at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> 18043562307- at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:944)
> 18043562376- at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:126)
> 18043562445- at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1686)
> 18043562535- at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1677)
> 18043562625- at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> 18043562709- at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> 18043562790- at java.lang.Thread.run(Thread.java:745)
> 18043562832-2015-04-01 16:23:08,882 INFO [AsyncDispatcher event handler] event.AsyncDispatcher: Exiting, bbye..
> 18043562932-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked
> 18043563023-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster: DAGAppMaster received a signal. Signaling TaskScheduler
> 18043563137-2015-04-01 16:23:08,885 INFO [Thread-1] rm.TaskSchedulerEventHandler: TaskScheduler notified that iSignalled was : true
> 18043563257-2015-04-01 16:23:08,899 INFO [Thread-1] history.HistoryEventHandler: Stopping HistoryEventHandler
> 18043563355-2015-04-01 16:23:08,900 INFO [Thread-1] recovery.RecoveryService: Stopping RecoveryService
> 18043563446-2015-04-01 16:23:08,900 INFO [Thread-1] recovery.RecoveryService: Closing Summary Stream
> 18043563535-2015-04-01 16:23:08,900 INFO [RecoveryEventHandlingThread] recovery.RecoveryService: EventQueue take interrupted. Returning
> 18043563659-2015-04-01 16:23:09,033 INFO [Thread-1] recovery.RecoveryService: Closing Output Stream for DAG dag_1426707664723_1086_1
> 18043563780-2015-04-01 16:23:09,062 INFO [Thread-1] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0
> 18043563891-2015-04-01 16:23:09,064 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000319
> 18043564052-2015-04-01 16:23:09,064 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn113-10.l42scl.hortonworks.com:45454
> 18043564185-2015-04-01 16:23:09,097 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000047
> 18043564346-2015-04-01 16:23:09,097 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn122-10.l42scl.hortonworks.com:45454
> 18043564479-2015-04-01 16:23:09,114 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000306
> 18043564640-2015-04-01 16:23:09,114 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn113-10.l42scl.hortonworks.com:45454
> 18043564773-2015-04-01 16:23:09,120 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000104
> 18043564934-2015-04-01 16:23:09,120 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn111-10.l42scl.hortonworks.com:45454
> 18043565067-2015-04-01 16:23:09,145 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000140
> 18043565228-2015-04-01 16:23:09,145 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn120-10.l42scl.hortonworks.com:45454
> 18043565361-2015-04-01 16:23:09,152 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000236
> 18043565522-2015-04-01 16:23:09,152 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn107-10.l42scl.hortonworks.com:45454
> 18043565655-2015-04-01 16:23:09,159 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000255
> 18043565816-2015-04-01 16:23:09,159 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn116-10.l42scl.hortonworks.com:45454
> 18043565949-2015-04-01 16:23:09,182 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000074
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)