You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Mostafa Mokhtar (JIRA)" <ji...@apache.org> on 2015/04/02 00:14:54 UTC
[jira] [Created] (TEZ-2263) Tez : Don't try to recover from a
failed commit
Mostafa Mokhtar created TEZ-2263:
------------------------------------
Summary: Tez : Don't try to recover from a failed commit
Key: TEZ-2263
URL: https://issues.apache.org/jira/browse/TEZ-2263
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Mostafa Mokhtar
Fix For: 0.5.0
Commit fails then Tez tries to recover which fails again.
{code}
5499174247-2015-04-01 16:23:20,600 INFO [main] app.RecoveryParser: Found summary file in attempt directory, summaryFile=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1/summary, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1
5499174696-2015-04-01 16:23:20,600 INFO [main] app.RecoveryParser: Using hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1 for recovering data from previous attempt
5499174963-2015-04-01 16:23:20,690 INFO [main] app.RecoveryParser: Parsing summary file, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1/summary, len=4024, lastModTime=1427919788998
5499175254-2015-04-01 16:23:20,786 INFO [main] app.RecoveryParser: Reached end of summary stream
5499175340-2015-04-01 16:23:21,087 INFO [main] app.RecoveryParser: Checking if DAG is in recoverable state, dagId=dag_1426707664723_1086_1
5499175468-2015-04-01 16:23:21,088 WARN [main] app.RecoveryParser: Found last inProgress DAG but not recoverable: dagId=dag_1426707664723_1086_1, dagCompleted=false
5499175622-2015-04-01 16:23:21,088 INFO [main] app.RecoveryParser: Trying to recover dag from recovery file, dagId=dag_1426707664723_1086_1, dataDir=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/1, intoCurrentDir=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2
5499176102-2015-04-01 16:23:21,091 INFO [main] app.RecoveryParser: Copying DAG data into Current Attempt directory, filePath=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2/dag_1426707664723_1086_1.recovery
5499176413-2015-04-01 16:23:21,211 INFO [main] app.RecoveryParser: Recovering from event, eventType=DAG_SUBMITTED, event=dagID=dag_1426707664723_1086_1, submitTime=1427917169723
5499176580-2015-04-01 16:23:21,309 INFO [main] app.DAGAppMaster: Generating DAG graphviz file, dagId=dag_1426707664723_1086_1, filePath=/grid/0/cluster/yarn/log/application_1426707664723_1086/container_1426707664723_1086_02_000001/dag_1426707664723_1086_1.dot
5499176829-2015-04-01 16:23:21,347 INFO [main] app.DAGAppMaster: Writing DAG plan to: /grid/0/cluster/yarn/log/application_1426707664723_1086/container_1426707664723_1086_02_000001/dag_1426707664723_1086_1-tez-dag.pb.txt
5499177039-2015-04-01 16:23:22,576 INFO [main] app.RecoveryParser: Finished copying data from previous attempt into current attempt
5499177160-2015-04-01 16:23:22,576 INFO [main] app.RecoveryParser: Trying to create data recovered flag file, filePath=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086/recovery/2/dataRecovered
5499177445-2015-04-01 16:23:22,601 INFO [main] app.DAGAppMaster: In Session mode. Waiting for DAG over RPC
5499177541-2015-04-01 16:23:22,601 INFO [main] app.DAGAppMaster: Found previous DAG in completed or non-recoverable state, dagId=dag_1426707664723_1086_1, isCompleted=false, isNonRecoverable=true, state=null, failureReason=DAG Commit was in progress, not recoverable, dagId=dag_1426707664723_1086_1
5499177829-2015-04-01 16:23:22,601 INFO [main] common.TezUtilsInternal: Redirecting log file based on addend: dag_1426707664723_1086_1
5499177953-
5499177954-LogType:syslog_dag_1426707664723_1086_1
5499177994-Log Upload Time:1-Apr-2015 16:24:30
5499178030-LogLength:521
5499178044-Log Contents:
5499178058-2015-04-01 16:23:22,604 INFO [main] impl.DAGImpl: Recovered DAG: dag_1426707664723_1086_1 finished with state: FAILED
5499178176-2015-04-01 16:23:22,604 INFO [main] impl.DAGImpl: dag_1426707664723_1086_1 transitioned from NEW to FAILED
5499178283-2015-04-01 16:23:22,604 INFO [AsyncDispatcher event handler] app.DAGAppMaster: DAG completed, dagId=dag_1426707664723_1086_1, dagState=FAILED
5499178425-2015-04-01 16:23:22,605 INFO [AsyncDispatcher event handler] common.TezUtilsInternal: Redirecting log file based on addend: dag_1426707664723_1086_1_post
5499178579-
5499178580-LogType:syslog_dag_1426707664723_1086_1_post
5499178625-Log Upload Time:1-Apr-2015 16:24:30
5499178661-LogLength:4021
5499178676-Log Contents:
5499178690-2015-04-01 16:23:22,605 INFO [AsyncDispatcher event handler] app.DAGAppMaster: Waiting for next DAG to be submitted.
5499178807-2015-04-01 16:24:01,681 INFO [IPC Server handler 0 on 53890] client.DAGClientHandler: Received message to shutdown AM
5499178925-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] rm.TaskSchedulerEventHandler: TaskScheduler notified that it should unregister from RM
5499179073-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: No current running DAG, shutting down the AM
5499179197-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: DAGAppMasterShutdownHandler invoked
5499179312-2015-04-01 16:24:01,682 INFO [IPC Server handler 0 on 53890] app.DAGAppMaster: Handling DAGAppMaster shutdown
5499179422-2015-04-01 16:24:01,683 INFO [AMShutdownThread] app.DAGAppMaster: Sleeping for 5 seconds before shutting down
5499179532-2015-04-01 16:24:04,151 INFO [HistoryEventHandlingThread] ats.ATSHistoryLoggingService: Event queue stats, eventsProcessedSinceLastUpdate=2, eventQueueSize=0
5499179690-2015-04-01 16:24:06,683 INFO [AMShutdownThread] app.DAGAppMaster: Calling stop for all the services
5499179790-2015-04-01 16:24:06,686 INFO [AMShutdownThread] history.HistoryEventHandler: Stopping HistoryEventHandler
5499179896-2015-04-01 16:24:06,686 INFO [AMShutdownThread] recovery.RecoveryService: Stopping RecoveryService
5499179995-2015-04-01 16:24:06,686 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0
5499180114-2015-04-01 16:24:06,686 INFO [RecoveryEventHandlingThread] recovery.RecoveryService: EventQueue take interrupted. Returning
5499180238-2015-04-01 16:24:06,692 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted
5499180367-2015-04-01 16:24:06,697 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=0, successfulDAGs=0, failedDAGs=1, killedDAGs=0
5499180589-, trackingURL=
5499180604-2015-04-01 16:24:06,713 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
5499180730-2015-04-01 16:24:06,819 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Successfully unregistered application from RM
5499180853-2015-04-01 16:24:06,821 INFO [AMShutdownThread] ipc.Server: Stopping server on 50998
5499180938-2015-04-01 16:24:06,821 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl: Interrupted while waiting for queue
5499181060:java.lang.InterruptedException
5499181091- at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
5499181228- at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
5499181346- at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
5499181426- at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
5499181551-2015-04-01 16:24:06,823 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder
5499181645-2015-04-01 16:24:06,822 INFO [IPC Server listener on 50998] ipc.Server: Stopping IPC Server listener on 50998
5499181755-2015-04-01 16:24:06,822 INFO [AMShutdownThread] ipc.Server: Stopping server on 53890
5499181840-2015-04-01 16:24:06,826 INFO [IPC Server listener on 53890] ipc.Server: Stopping IPC Server listener on 53890
5499181950-2015-04-01 16:24:06,826 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder
5499182044-2015-04-01 16:24:06,828 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked
5499182135-2015-04-01 16:24:06,828 INFO [Thread-1] app.DAGAppMaster: The shutdown handler is still running, waiting for it to complete
5499182259-2015-04-01 16:24:06,839 WARN [AMShutdownThread] app.DAGAppMaster: Failed to delete tez scratch data dir, path=hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/mmokhtar/_tez_session_dir/8b149f3c-3947-4b34-a5e2-92657aa68e96/.tez/application_1426707664723_1086
5499182521-2015-04-01 16:24:06,839 INFO [AMShutdownThread] app.DAGAppMaster: Exiting DAGAppMaster..GoodBye!
5499182618-2015-04-01 16:24:06,839 INFO [Thread-1] app.DAGAppMaster: The shutdown handler has completed
5499182711-
5499182712-
5499182713-
5499182714-Container: container_1426707664723_1086_01_000304 on cn110-10.l42scl.hortonworks.com_45454
5499182805-============================================================================================
5499182898-LogType:stderr
5499182913-Log Upload Time:1-Apr-2015 16:24:30
5499182949-LogLength:0
5499182961-Log Contents:
5499182975-
5499182976-LogType:stdout
5499182991-Log Upload Time:1-Apr-2015 16:24:30
5499183027-LogLength:81042
5499183043-Log Contents:
5499183057-9.076: [GC [PSYoungGen: 415194K->108031K(756736K)] 1463770K->1182454K(7674880K), 0.1355580 secs] [Times: user=0.59 sys=0.13, real=0.13 secs]
5499183199-16.885: [GC [PSYoungGen: 446979K->70470K(756736K)] 1521402K->1144901K(7674880K), 0.1141880 secs] [Times: user=0.41 sys=0.08, real=0.11 secs]
5499183341-32.610: [GC [PSYoungGen: 401811K->23054K(756736K)] 1476242K->1097493K(7674880K), 0.0198940 secs] [Times: user=0.26 sys=0.00, real=0.02 secs]
5499183483-37.397: [GC [PSYoungGen: 354557K->108009K(756736K)] 2477572K->2246623K(7674880K), 0.0591560 secs] [Times: user=0.82 sys=0.06, real=0.06 secs]
5499183626-42.607: [GC [PSYoungGen: 434535K->15928K(649216K)] 2573148K->2154549K(7567360K), 0.0639210 secs] [Times: user=0.45 sys=0.03, real=0.06 secs]
5499183768-47.641: [GC [PSYoungGen: 553804K->22893K(689152K)] 3741001K->3210098K(7607296K), 0.0727270 secs] [Times: user=0.55 sys=0.06, real=0.07 secs]
5499183910-53.107: [GC [PSYoungGen: 343874K->137457K(646656K)] 4579655K->4384105K(7564800K), 0.1012020 secs] [Times: user=0.50 sys=0.14, real=0.10 secs]
5499184053-62.586: [GC [PSYoungGen: 484074K->89878K(676864K)] 4730722K->4336778K(7595008K), 0.0985300 secs] [Times: user=0.56 sys=0.08, real=0.10 secs]
5499184195-76.005: [GC [PSYoungGen: 449294K->16827K(668672K)] 4696194K->4274117K(7586816K), 0.0315610 secs] [Times: user=0.36 sys=0.03, real=0.03 secs]
5499184337-79.777: [GC [PSYoungGen: 211179K->19599K(680448K)] 5517045K->5332144K(7598592K), 0.0304100 secs] [Times: user=0.37 sys=0.02, real=0.03 secs]
5499184479-81.315: [GC [PSYoungGen: 150008K->78090K(677888K)] 5462554K->5399597K(7596032K), 0.0293570 secs] [Times: user=0.39 sys=0.02, real=0.03 secs]
5499184621-82.455: [GC [PSYoungGen: 237557K->512K(687616K)] 5559064K->5324779K(7605760K), 0.0384990 secs] [Times: user=0.34 sys=0.02, real=0.04 secs]
5499184761-84.067: [GC [PSYoungGen: 210827K->14117K(687616K)] 6583671K->6387049K(7605760K), 0.0517180 secs] [Times: user=0.66 sys=0.01, real=0.05 secs]
5499184903-88.416: [GC [PSYoungGen: 268920K->15351K(696320K)] 6641851K->6394989K(7614464K), 0.0787950 secs] [Times: user=0.51 sys=0.02, real=0.08 secs]
5499185045-101.043: [GC [PSYoungGen: 376721K->448K(691200K)] 6756359K->6387846K(7609344K), 0.0282280 secs] [Times: user=0.44 sys=0.02, real=0.03 secs]
5499185186-103.105: [GC [PSYoungGen: 82965K->13643K(705536K)] 6470363K->6401129K(7623680K), 0.1482160 secs] [Times: user=0.53 sys=0.00, real=0.15 secs]
5499185328-103.253: [GC [PSYoungGen: 13643K->0K(699392K)] 6401129K->6400834K(7617536K), 0.0805200 secs] [Times: user=0.56 sys=0.02, real=0.08 secs]
5499185466-103.334: [Full GC [PSYoungGen: 0K->0K(699392K)] [ParOldGen: 6400834K->26007K(6918144K)] 6400834K->26007K(7617536K) [PSPermGen: 33081K->33057K(66560K)], 0.4079400 secs] [Times: user=1.08 sys=0.17, real=0.41 secs]
5499185679-108.478: [GC [PSYoungGen: 279515K->23409K(718848K)] 1354098K->1097992K(7636992K), 0.0143960 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
5499185822-121.280: [GC [PSYoungGen: 322641K->279K(709632K)] 1397224K->1082530K(7627776K), 0.0187460 secs] [Times: user=0.10 sys=0.00, real=0.01 secs]
5499185963-126.595: [GC [PSYoungGen: 345663K->20073K(731648K)] 2476491K->2150964K(7649792K), 0.0305890 secs] [Times: user=0.19 sys=0.00, real=0.03 secs]
5499186106-141.120: [GC [PSYoungGen: 596191K->14241K(723456K)] 3775658K->3200292K(7641600K), 0.0189990 secs] [Times: user=0.27 sys=0.00, real=0.02 secs]
--
18043554745-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 49
18043554860-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 44
18043554975-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 45
18043555090-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 50
18043555205-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 25
18043555320-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 17
18043555435-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 23
18043555550-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 24
18043555665-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 21
18043555780-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 22
18043555895-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 38
18043556014-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 20
18043556129-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 31
18043556248-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 37
18043556367-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 29
18043556482-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 30
18043556601-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 27
18043556716-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 28
18043556831-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 3
18043556949-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 26
18043557064-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 2
18043557182-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 32
18043557297-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 36
18043557416-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 35
18043557535-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 34
18043557654-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 33
18043557769-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 1
18043557883-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 39
18043557998-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 5
18043558112-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 4
18043558226-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 43
18043558341-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 53
18043558456-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 42
18043558571-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 54
18043558686-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 41
18043558801-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 51
18043558916-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 14
18043559031-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 7
18043559149-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 13
18043559264-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 6
18043559382-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 16
18043559497-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 19
18043559612-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 18
18043559727-2015-04-01 16:23:08,508 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Map 15
18043559842-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No exclusive output committers for vertex: Reducer 12
18043559971-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 10
18043560090-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 11
18043560209-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 8
18043560327-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler] impl.DAGImpl: No output committers for vertex: Reducer 9
18043560445-2015-04-01 16:23:08,857 FATAL [AsyncDispatcher event handler] event.AsyncDispatcher: Error in dispatcher thread
18043560557:org.apache.tez.common.counters.LimitExceededException: Too many counters: 1201 max=1200
18043560645- at org.apache.tez.common.counters.Limits.checkCounters(Limits.java:87)
18043560717- at org.apache.tez.common.counters.Limits.incrCounters(Limits.java:94)
18043560788- at org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:75)
18043560885- at org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
18043560986- at org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:103)
18043561085- at org.apache.tez.common.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:198)
18043561188- at org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:363)
18043561283- at org.apache.tez.dag.app.dag.impl.DAGImpl.incrTaskCounters(DAGImpl.java:598)
18043561362- at org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:588)
18043561439- at org.apache.tez.dag.app.dag.impl.DAGImpl.logJobHistoryFinishedEvent(DAGImpl.java:994)
18043561528- at org.apache.tez.dag.app.dag.impl.DAGImpl.finished(DAGImpl.java:1135)
18043561600- at org.apache.tez.dag.app.dag.impl.DAGImpl.checkDAGForCompletion(DAGImpl.java:1048)
18043561685- at org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1708)
18043561785- at org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1665)
18043561885- at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
18043562001- at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
18043562097- at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
18043562190- at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
18043562307- at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:944)
18043562376- at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:126)
18043562445- at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1686)
18043562535- at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1677)
18043562625- at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
18043562709- at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
18043562790- at java.lang.Thread.run(Thread.java:745)
18043562832-2015-04-01 16:23:08,882 INFO [AsyncDispatcher event handler] event.AsyncDispatcher: Exiting, bbye..
18043562932-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked
18043563023-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster: DAGAppMaster received a signal. Signaling TaskScheduler
18043563137-2015-04-01 16:23:08,885 INFO [Thread-1] rm.TaskSchedulerEventHandler: TaskScheduler notified that iSignalled was : true
18043563257-2015-04-01 16:23:08,899 INFO [Thread-1] history.HistoryEventHandler: Stopping HistoryEventHandler
18043563355-2015-04-01 16:23:08,900 INFO [Thread-1] recovery.RecoveryService: Stopping RecoveryService
18043563446-2015-04-01 16:23:08,900 INFO [Thread-1] recovery.RecoveryService: Closing Summary Stream
18043563535-2015-04-01 16:23:08,900 INFO [RecoveryEventHandlingThread] recovery.RecoveryService: EventQueue take interrupted. Returning
18043563659-2015-04-01 16:23:09,033 INFO [Thread-1] recovery.RecoveryService: Closing Output Stream for DAG dag_1426707664723_1086_1
18043563780-2015-04-01 16:23:09,062 INFO [Thread-1] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=0
18043563891-2015-04-01 16:23:09,064 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000319
18043564052-2015-04-01 16:23:09,064 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn113-10.l42scl.hortonworks.com:45454
18043564185-2015-04-01 16:23:09,097 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000047
18043564346-2015-04-01 16:23:09,097 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn122-10.l42scl.hortonworks.com:45454
18043564479-2015-04-01 16:23:09,114 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000306
18043564640-2015-04-01 16:23:09,114 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn113-10.l42scl.hortonworks.com:45454
18043564773-2015-04-01 16:23:09,120 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000104
18043564934-2015-04-01 16:23:09,120 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn111-10.l42scl.hortonworks.com:45454
18043565067-2015-04-01 16:23:09,145 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000140
18043565228-2015-04-01 16:23:09,145 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn120-10.l42scl.hortonworks.com:45454
18043565361-2015-04-01 16:23:09,152 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000236
18043565522-2015-04-01 16:23:09,152 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn107-10.l42scl.hortonworks.com:45454
18043565655-2015-04-01 16:23:09,159 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000255
18043565816-2015-04-01 16:23:09,159 INFO [Thread-1] impl.ContainerManagementProtocolProxy: Opening proxy : cn116-10.l42scl.hortonworks.com:45454
18043565949-2015-04-01 16:23:09,182 INFO [Thread-1] launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1426707664723_1086_01_000074
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)