You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Hudson (JIRA)" <ji...@apache.org> on 2015/09/18 21:27:04 UTC
[jira] [Commented] (AMBARI-13148) Ambari DB schema upgrade command
hung on console even though schema upgrade is success
[ https://issues.apache.org/jira/browse/AMBARI-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876203#comment-14876203 ]
Hudson commented on AMBARI-13148:
---------------------------------
SUCCESS: Integrated in Ambari-trunk-Commit #3465 (See [https://builds.apache.org/job/Ambari-trunk-Commit/3465/])
AMBARI-13148. Ambari DB schema upgrade command hung on console even though schema upgrade is success (dlysnichenko) (dlysnichenko: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=ad20650cdcb3415c5231228bceefa085c827a9cb)
* ambari-server/src/main/java/org/apache/ambari/server/security/encryption/InMemoryCredentialStoreService.java
> Ambari DB schema upgrade command hung on console even though schema upgrade is success
> --------------------------------------------------------------------------------------
>
> Key: AMBARI-13148
> URL: https://issues.apache.org/jira/browse/AMBARI-13148
> Project: Ambari
> Issue Type: Bug
> Reporter: Dmitry Lysnichenko
> Assignee: Dmitry Lysnichenko
> Attachments: AMBARI-13148.patch
>
>
> Observed this issue on two configs
> Config #1 -- Ambari server upgrade from 1.6.1 to 2.1.2-262 on Suse 11 SP3
> Config #2 -- Ambari server upgrade from 2.0.2 to 2.1.2-262 on Ubuntu 12
> Ambari DB: Embedded Postgres
> The upgrade command was hung and did not come out of the console:
> vsharma-u21todalm10-re-5:~ # ambari-server upgrade
> Using python /usr/bin/python2.6
> Upgrading ambari-server
> Updating properties in ambari.properties ...
> WARNING: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings
> Fixing database objects owner
> Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y
> Upgrading database schema
> When looking at ambari-server.log - the upgrade is a success (see attached). See below for process and jstack output of the upgrade process
> vsharma-u21todalm10-re-5:/tmp/dbdumps # ps -ef | grep ambari
> root 22879 22756 0 17:23 pts/1 00:00:00 bash /usr/sbin/ambari-server upgrade
> root 22888 22879 0 17:23 pts/1 00:00:00 /usr/bin/python2.6 /usr/sbin/ambari-server.py upgrade
> root 23071 22888 4 17:23 pts/1 00:00:33 /usr/jdk64/jdk1.7.0_67/bin/java -cp /etc/ambari-server/conf:/usr/lib/ambari-server/* org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1
> postgres 23085 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48478) idle
> postgres 23086 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48479) idle
> postgres 23087 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48480) idle
> postgres 23088 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48481) idle
> postgres 23089 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48482) idle
> postgres 23090 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48483) idle
> postgres 23091 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48484) idle
> postgres 23092 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48485) idle
> postgres 23093 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48486) idle
> postgres 23094 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48487) idle
> postgres 23095 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48488) idle
> postgres 23096 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48489) idle
> postgres 23097 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48490) idle
> postgres 23098 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48491) idle
> postgres 23099 6255 0 17:23 ? 00:00:02 postgres: ambari ambari 127.0.0.1(48492) idle
> postgres 23100 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48493) idle
> postgres 23101 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48494) idle
> postgres 23102 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48495) idle
> postgres 23103 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48496) idle
> postgres 23104 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48497) idle
> postgres 23105 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48498) idle
> postgres 23108 6255 0 17:23 ? 00:00:00 postgres: ambari ambari 127.0.0.1(48499) idle
> root 23214 22790 0 17:36 pts/2 00:00:00 grep ambari
> vsharma-u21todalm10-re-5:/tmp/dbdumps # cd /usr/jdk64/jdk1.7.0_67/bin/
> vsharma-u21todalm10-re-5:/usr/jdk64/jdk1.7.0_67/bin # ./jstack 23071
> 2015-09-16 17:37:14
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):
> "Attach Listener" daemon prio=10 tid=0x00000000014ed000 nid=0x5ac5 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "DestroyJavaVM" prio=10 tid=0x00007f8fec53f000 nid=0x5a20 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "InMemoryCredentialStoreService active cleanup timer" prio=10 tid=0x00007f8fecbfd000 nid=0x5a4c waiting on condition [0x00007f8ffa28a000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007cfa39908> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "com.google.inject.internal.util.$Finalizer" daemon prio=10 tid=0x00007f8fec18a800 nid=0x5a2b in Object.wait() [0x00007f8ffaa19000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000007cfa3a6f8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000007cfa3a6f8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
> "Service Thread" daemon prio=10 tid=0x00007f8ff4029800 nid=0x5a29 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f8ff4027000 nid=0x5a28 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f8ff4024000 nid=0x5a27 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=10 tid=0x00007f8ff4022000 nid=0x5a26 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=10 tid=0x0000000000675000 nid=0x5a25 in Object.wait() [0x00007f8ffbb5b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000707c613e8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x0000000707c613e8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" daemon prio=10 tid=0x000000000066d800 nid=0x5a24 in Object.wait() [0x00007f8ffbc5c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000707c614d0> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> - locked <0x0000000707c614d0> (a java.lang.ref.Reference$Lock)
> "VM Thread" prio=10 tid=0x0000000000669000 nid=0x5a23 runnable
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000000623800 nid=0x5a21 runnable
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000000625800 nid=0x5a22 runnable
> "VM Periodic Task Thread" prio=10 tid=0x00007f8ff4034800 nid=0x5a2a waiting on condition
> JNI global references: 388
> ========================================
> After I manually killed the above process and started ambari-server, ambari server started successfully
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)