You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/08/13 10:00:01 UTC
[jira] [Updated] (AMBARI-24462) ambari-server upgrade stuck with
NPE
[ https://issues.apache.org/jira/browse/AMBARI-24462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated AMBARI-24462:
------------------------------------
Labels: pull-request-available (was: )
> ambari-server upgrade stuck with NPE
> ------------------------------------
>
> Key: AMBARI-24462
> URL: https://issues.apache.org/jira/browse/AMBARI-24462
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.7.0
> Reporter: Attila Magyar
> Assignee: Attila Magyar
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.7.1
>
>
> ====================================
> Ambari 2.6.2
> HDP 2.6.5.0-292
> Cluster Details : https://github.com/hortonworks/HCube#stanley-hotel-hdp-26
> Please login with okta credential to these cluster machines
> https://hcube1-1n01.eng.hortonworks.com:8443/
> ====================================
> This is cluster is being upgraded to Ambari 2.7.1 and the schema upgrade step is hung.
> {code}
> root@hcube1-1n01 ~]# ambari-server upgrade
> Using python /usr/bin/python
> Upgrading ambari-server
> INFO: Upgrade Ambari Server
> INFO: Updating Ambari Server properties in ambari.properties ...
> INFO: Updating Ambari Server properties in ambari-env.sh ...
> INFO: Original file ambari-env.sh kept
> WARNING: Original file krb5JAASLogin.conf kept
> INFO: File krb5JAASLogin.conf updated.
> INFO: Fixing database objects owner
> Ambari Server configured for Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y
> INFO: Upgrading database schema
> {code}
> Below exception is noticed in ambari-server.log
> {code}
> 2018-08-10 06:57:50,795 ERROR [main] AbstractUpgradeCatalog:375 - Error in transaction
> java.lang.NullPointerException
> at org.apache.ambari.server.controller.KerberosHelperImpl.addIdentities(KerberosHelperImpl.java:1617)
> at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponents(AbstractPrepareKerberosServerAction.java:184)
> at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponentHosts(AbstractPrepareKerberosServerAction.java:94)
> at org.apache.ambari.server.upgrade.UpgradeCatalog270$3.run(UpgradeCatalog270.java:1637)
> at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:367)
> at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1633)
> at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1060)
> at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
> at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
> at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:457)
> 2018-08-10 06:57:50,799 ERROR [main] SchemaUpgradeHelper:239 - Upgrade failed.
> org.apache.ambari.server.AmbariException: Failed to upgrade kerberos tables
> at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1644)
> at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1060)
> at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
> at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
> at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:457)
> Caused by: java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:379)
> at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1633)
> ... 4 more
> Caused by: java.lang.NullPointerException
> at org.apache.ambari.server.controller.KerberosHelperImpl.addIdentities(KerberosHelperImpl.java:1617)
> at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponents(AbstractPrepareKerberosServerAction.java:184)
> at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponentHosts(AbstractPrepareKerberosServerAction.java:94)
> at org.apache.ambari.server.upgrade.UpgradeCatalog270$3.run(UpgradeCatalog270.java:1637)
> at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:367)
> ... 5 more
> {code}
> Attached is the complete ambari-server logs
> Below is the jstack dump of schemaupgradehelper
> {code}
> [root@hcube1-1n01 ambari-server]# /opt/java/jdk1.8.0_152/bin/jstack -l 8239
> 2018-08-10 07:10:00
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.152-b16 mixed mode):
> "Attach Listener" #40 daemon prio=9 os_prio=0 tid=0x00007f9a6c001000 nid=0x3afa waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "DestroyJavaVM" #39 prio=5 os_prio=0 tid=0x00007f9b6c00d000 nid=0x2030 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "FileWatchdog:ambari.properties" #38 daemon prio=5 os_prio=0 tid=0x00007f9b6d684000 nid=0x2c19 waiting on condition [0x00007f9b2e9d4000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
> Locked ownable synchronizers:
> - None
> "pool-6-thread-1" #37 prio=5 os_prio=0 tid=0x00007f9a10006800 nid=0x210f waiting on condition [0x00007f9b2eed9000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000080dc3270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "pool-5-thread-1" #36 prio=5 os_prio=0 tid=0x00007f9b6d497800 nid=0x2108 waiting on condition [0x00007f9b2edd8000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000080dc2fe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "pool-3-thread-1" #35 prio=5 os_prio=0 tid=0x00007f9b6d0cb000 nid=0x2107 waiting on condition [0x00007f9b2f2dd000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000080dc2b38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "server-action-executor-cache-timer" #23 daemon prio=5 os_prio=0 tid=0x00007f9b6da29000 nid=0x20d8 in Object.wait() [0x00007f9b34120000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000080f0b6e0> (a java.util.TaskQueue)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> - locked <0x0000000080f0b6e0> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:505)
> Locked ownable synchronizers:
> - None
> "InMemoryCredentialStore active cleanup timer" #22 daemon prio=5 os_prio=0 tid=0x00007f9b6d541800 nid=0x20cd waiting on condition [0x00007f9b35021000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000080f2c750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - None
> "Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007f9b6c107800 nid=0x205c runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f9b6c104800 nid=0x205b waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f9b6c102800 nid=0x205a waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f9b6c100000 nid=0x2059 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f9b6c0fe000 nid=0x2058 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f9b6c0fb800 nid=0x2057 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f9800 nid=0x2056 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f7800 nid=0x2054 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f5000 nid=0x2053 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f3000 nid=0x2052 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f0800 nid=0x2051 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ee800 nid=0x2050 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ec000 nid=0x204f waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ea000 nid=0x204e waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e8000 nid=0x204d waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e5000 nid=0x204c waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e3800 nid=0x204b runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> Locked ownable synchronizers:
> - None
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9b6c0af800 nid=0x204a in Object.wait() [0x00007f9b3d0e2000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000008020b148> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x000000008020b148> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> Locked ownable synchronizers:
> - None
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9b6c0ab000 nid=0x2049 in Object.wait() [0x00007f9b3d1e3000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x000000008020c698> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x000000008020c698> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> Locked ownable synchronizers:
> - None
> "VM Thread" os_prio=0 tid=0x00007f9b6c0a3800 nid=0x2048 runnable
> "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f9b6c022800 nid=0x2031 runnable
> "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f9b6c024000 nid=0x2032 runnable
> "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f9b6c026000 nid=0x2033 runnable
> "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f9b6c028000 nid=0x2034 runnable
> "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f9b6c029800 nid=0x2035 runnable
> "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02b800 nid=0x2036 runnable
> "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02d000 nid=0x2037 runnable
> "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02f000 nid=0x2038 runnable
> "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f9b6c031000 nid=0x2039 runnable
> "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f9b6c032800 nid=0x203a runnable
> "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f9b6c034800 nid=0x203b runnable
> "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f9b6c036800 nid=0x203c runnable
> "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f9b6c038000 nid=0x203d runnable
> "GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03a000 nid=0x203e runnable
> "GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03c000 nid=0x203f runnable
> "GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03d800 nid=0x2040 runnable
> "GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03f800 nid=0x2041 runnable
> "GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007f9b6c041800 nid=0x2042 runnable
> "GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007f9b6c043000 nid=0x2043 runnable
> "GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007f9b6c045000 nid=0x2044 runnable
> "GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007f9b6c046800 nid=0x2045 runnable
> "GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007f9b6c048800 nid=0x2046 runnable
> "GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007f9b6c04a800 nid=0x2047 runnable
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f9b6c10c800 nid=0x205d waiting on condition
> JNI global references: 328
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)