You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Attila Magyar (JIRA)" <ji...@apache.org> on 2018/08/13 09:54:00 UTC
[jira] [Created] (AMBARI-24462) ambari-server upgrade stuck with
NPE
Attila Magyar created AMBARI-24462:
--------------------------------------
Summary: ambari-server upgrade stuck with NPE
Key: AMBARI-24462
URL: https://issues.apache.org/jira/browse/AMBARI-24462
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.7.0
Reporter: Attila Magyar
Assignee: Attila Magyar
Fix For: 2.7.1
====================================
Ambari 2.6.2
HDP 2.6.5.0-292
Cluster Details : https://github.com/hortonworks/HCube#stanley-hotel-hdp-26
Please login with okta credential to these cluster machines
https://hcube1-1n01.eng.hortonworks.com:8443/
====================================
This is cluster is being upgraded to Ambari 2.7.1 and the schema upgrade step is hung.
{code}
root@hcube1-1n01 ~]# ambari-server upgrade
Using python /usr/bin/python
Upgrading ambari-server
INFO: Upgrade Ambari Server
INFO: Updating Ambari Server properties in ambari.properties ...
INFO: Updating Ambari Server properties in ambari-env.sh ...
INFO: Original file ambari-env.sh kept
WARNING: Original file krb5JAASLogin.conf kept
INFO: File krb5JAASLogin.conf updated.
INFO: Fixing database objects owner
Ambari Server configured for Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y
INFO: Upgrading database schema
{code}
Below exception is noticed in ambari-server.log
{code}
2018-08-10 06:57:50,795 ERROR [main] AbstractUpgradeCatalog:375 - Error in transaction
java.lang.NullPointerException
at org.apache.ambari.server.controller.KerberosHelperImpl.addIdentities(KerberosHelperImpl.java:1617)
at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponents(AbstractPrepareKerberosServerAction.java:184)
at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponentHosts(AbstractPrepareKerberosServerAction.java:94)
at org.apache.ambari.server.upgrade.UpgradeCatalog270$3.run(UpgradeCatalog270.java:1637)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:367)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1633)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1060)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:457)
2018-08-10 06:57:50,799 ERROR [main] SchemaUpgradeHelper:239 - Upgrade failed.
org.apache.ambari.server.AmbariException: Failed to upgrade kerberos tables
at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1644)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1060)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:457)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:379)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1633)
... 4 more
Caused by: java.lang.NullPointerException
at org.apache.ambari.server.controller.KerberosHelperImpl.addIdentities(KerberosHelperImpl.java:1617)
at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponents(AbstractPrepareKerberosServerAction.java:184)
at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponentHosts(AbstractPrepareKerberosServerAction.java:94)
at org.apache.ambari.server.upgrade.UpgradeCatalog270$3.run(UpgradeCatalog270.java:1637)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:367)
... 5 more
{code}
Attached is the complete ambari-server logs
Below is the jstack dump of schemaupgradehelper
{code}
[root@hcube1-1n01 ambari-server]# /opt/java/jdk1.8.0_152/bin/jstack -l 8239
2018-08-10 07:10:00
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.152-b16 mixed mode):
"Attach Listener" #40 daemon prio=9 os_prio=0 tid=0x00007f9a6c001000 nid=0x3afa waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"DestroyJavaVM" #39 prio=5 os_prio=0 tid=0x00007f9b6c00d000 nid=0x2030 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"FileWatchdog:ambari.properties" #38 daemon prio=5 os_prio=0 tid=0x00007f9b6d684000 nid=0x2c19 waiting on condition [0x00007f9b2e9d4000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
Locked ownable synchronizers:
- None
"pool-6-thread-1" #37 prio=5 os_prio=0 tid=0x00007f9a10006800 nid=0x210f waiting on condition [0x00007f9b2eed9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080dc3270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"pool-5-thread-1" #36 prio=5 os_prio=0 tid=0x00007f9b6d497800 nid=0x2108 waiting on condition [0x00007f9b2edd8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080dc2fe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"pool-3-thread-1" #35 prio=5 os_prio=0 tid=0x00007f9b6d0cb000 nid=0x2107 waiting on condition [0x00007f9b2f2dd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080dc2b38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"server-action-executor-cache-timer" #23 daemon prio=5 os_prio=0 tid=0x00007f9b6da29000 nid=0x20d8 in Object.wait() [0x00007f9b34120000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000080f0b6e0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000000080f0b6e0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"InMemoryCredentialStore active cleanup timer" #22 daemon prio=5 os_prio=0 tid=0x00007f9b6d541800 nid=0x20cd waiting on condition [0x00007f9b35021000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000080f2c750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007f9b6c107800 nid=0x205c runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f9b6c104800 nid=0x205b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f9b6c102800 nid=0x205a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f9b6c100000 nid=0x2059 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f9b6c0fe000 nid=0x2058 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f9b6c0fb800 nid=0x2057 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f9800 nid=0x2056 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f7800 nid=0x2054 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f5000 nid=0x2053 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f3000 nid=0x2052 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f0800 nid=0x2051 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ee800 nid=0x2050 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ec000 nid=0x204f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ea000 nid=0x204e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e8000 nid=0x204d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e5000 nid=0x204c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e3800 nid=0x204b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9b6c0af800 nid=0x204a in Object.wait() [0x00007f9b3d0e2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000008020b148> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000008020b148> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9b6c0ab000 nid=0x2049 in Object.wait() [0x00007f9b3d1e3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000008020c698> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x000000008020c698> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f9b6c0a3800 nid=0x2048 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f9b6c022800 nid=0x2031 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f9b6c024000 nid=0x2032 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f9b6c026000 nid=0x2033 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f9b6c028000 nid=0x2034 runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f9b6c029800 nid=0x2035 runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02b800 nid=0x2036 runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02d000 nid=0x2037 runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02f000 nid=0x2038 runnable
"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f9b6c031000 nid=0x2039 runnable
"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f9b6c032800 nid=0x203a runnable
"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f9b6c034800 nid=0x203b runnable
"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f9b6c036800 nid=0x203c runnable
"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f9b6c038000 nid=0x203d runnable
"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03a000 nid=0x203e runnable
"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03c000 nid=0x203f runnable
"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03d800 nid=0x2040 runnable
"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03f800 nid=0x2041 runnable
"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007f9b6c041800 nid=0x2042 runnable
"GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007f9b6c043000 nid=0x2043 runnable
"GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007f9b6c045000 nid=0x2044 runnable
"GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007f9b6c046800 nid=0x2045 runnable
"GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007f9b6c048800 nid=0x2046 runnable
"GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007f9b6c04a800 nid=0x2047 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f9b6c10c800 nid=0x205d waiting on condition
JNI global references: 328
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)