You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Constantine Yarovoy <ky...@gmail.com> on 2015/12/18 16:41:48 UTC
Re: Ambari server start takes too long
Jonathan, please check my jstack dump during startup (which lasts at about
4-5 minutes):
[root@kk-ambari bin]# ./jps -l
16691 sun.tools.jps.Jps
16650 org.apache.ambari.server.controller.AmbariServer
[root@kk-ambari bin]# ./jstack -F -l 16650
Attaching to process ID 16650, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.40-b25
Deadlock Detection:
No deadlocks found.
Thread 16664: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Interpreted
frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Interpreted
frame)
- com.google.inject.internal.util.$Finalizer.run() @bci=5, line=114
(Interpreted frame)
Locked ownable synchronizers:
- None
Thread 16659: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 16658: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 16657: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Interpreted
frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Interpreted
frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209
(Interpreted frame)
Locked ownable synchronizers:
- None
Thread 16656: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=36, line=157
(Interpreted frame)
Locked ownable synchronizers:
- None
Thread 16651: (state = IN_NATIVE)
- java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Interpreted
frame)
- java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255
(Interpreted frame)
-
sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(byte[])
@bci=19, line=539 (Interpreted frame)
- sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4,
line=144 (Interpreted frame)
- sun.security.provider.SecureRandom$SeederHolder.<clinit>() @bci=20,
line=203 (Interpreted frame)
- sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=21,
line=221 (Interpreted frame)
- java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=468
(Interpreted frame)
- org.snmp4j.security.Salt.<init>() @bci=17, line=55 (Interpreted frame)
- org.snmp4j.security.Salt.getInstance() @bci=10, line=80 (Interpreted
frame)
- org.snmp4j.security.PrivDES.<init>() @bci=5, line=57 (Interpreted frame)
- org.snmp4j.security.SecurityProtocols.addDefaultProtocols() @bci=362,
line=152 (Interpreted frame)
- org.snmp4j.Snmp.initMessageDispatcher() @bci=61, line=225 (Interpreted
frame)
- org.snmp4j.Snmp.<init>(org.snmp4j.TransportMapping) @bci=5, line=251
(Interpreted frame)
-
org.apache.ambari.server.notifications.dispatchers.SNMPDispatcher.<init>()
@bci=12, line=92 (Interpreted frame)
-
sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor,
java.lang.Object[]) @bci=0 (Interpreted frame)
-
sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[])
@bci=85, line=62 (Interpreted frame)
-
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[])
@bci=5, line=45 (Compiled frame)
- java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79,
line=422 (Compiled frame)
- java.lang.Class.newInstance() @bci=138, line=442 (Interpreted frame)
-
org.apache.ambari.server.controller.ControllerModule.bindNotificationDispatchers()
@bci=134, line=560 (Interpreted frame)
- org.apache.ambari.server.controller.ControllerModule.configure()
@bci=543, line=341 (Interpreted frame)
- com.google.inject.AbstractModule.configure(com.google.inject.Binder)
@bci=31, line=59 (Interpreted frame)
-
com.google.inject.spi.Elements$RecordingBinder.install(com.google.inject.Module)
@bci=31, line=223 (Interpreted frame)
- com.google.inject.spi.Elements.getElements(com.google.inject.Stage,
java.lang.Iterable) @bci=40, line=101 (Interpreted frame)
-
com.google.inject.internal.InjectorShell$Builder.build(com.google.inject.internal.Initializer,
com.google.inject.internal.ProcessedBindingData,
com.google.inject.internal.util.$Stopwatch,
com.google.inject.internal.Errors) @bci=99, line=133 (Interpreted frame)
- com.google.inject.internal.InternalInjectorCreator.build() @bci=48,
line=103 (Interpreted frame)
- com.google.inject.Guice.createInjector(com.google.inject.Stage,
java.lang.Iterable) @bci=15, line=95 (Interpreted frame)
- com.google.inject.Guice.createInjector(java.lang.Iterable) @bci=4,
line=72 (Interpreted frame)
- com.google.inject.Guice.createInjector(com.google.inject.Module[])
@bci=4, line=62 (Interpreted frame)
-
org.apache.ambari.server.controller.AmbariServer.main(java.lang.String[])
@bci=14, line=706 (Interpreted frame)
Locked ownable synchronizers:
- None
As far as I understand, it gets stuck waiting for entropy to build up ?
I don't think it's a bug, because I'm using jdk 1.8.
Any workaround ? I'm launching Ambari server in Centos 7 os running on
Openstack VM.
2015-11-16 20:58 GMT+02:00 Jonathan Hurley <jh...@hortonworks.com>:
> What this step is doing is loading classes which match an interface and
> binding them as individual alert dispatchers in Guice. I haven’t
> experienced any slowdown starting Ambari server - usually starts up in
> about 10 seconds total. Can you provide a jstack dump during your startup
> so we can see what the various threads are doing? I’m mainly concerned with
> the main Ambari thread that would be initializing this stuff.
>
> > On Nov 16, 2015, at 3:52 AM, Constantine Yarovoy <ky...@gmail.com>
> wrote:
> >
> > Hi all.
> >
> > I'm developing my own stack for Ambari and I often need to change
> master/slave component code in Python. And I have 2 questions regarding
> this:
> >
> > 1. What is the fastest way to make Ambari understand that stack has
> changed and to use updated code ? The only way it works for me now is to
> restart server (service ambari-server restart)
> >
> > Is it possible to do it any other way without restarting the server ?
> >
> > 2. Ambari server start procedure really takes too long. I'm using Centos
> 7 and after starting the service it takes at about 4-8 minutes for it to
> actually bind to port 8080 so that web ui becomes available. Tailing -f
> ambari-server.log I've notices that the biggest delay during start is on
> this step:
> >
> > 16 Nov 2015 08:47:13,392 INFO [main] ControllerModule:560 - Binding and
> registering notification dispatcher class
> org.apache.ambari.server.notifications.dispatchers.AlertScriptDispatcher
> >
> > Maybe someone experienced the same behavior and there is a way to speed
> up this step?
> >
> > Thanks in advance.
> >
> > --
> > Kostiantyn Yarovyi
> >
>
>
--
Regards,
Kostiantyn Yarovyi
CIO, ITIL Expert, Cisco CCNP, Microsoft MCITP
+38 (099) 148-96-28 | skype:kyarovoy
Re: Ambari server start takes too long
Posted by Jonathan Hurley <jh...@hortonworks.com>.
This is definitely a problem building up entropy for "true" randomness in /dev/random. You can try the workaround from Oracle:
$JAVA_HOME/jre/lib/security/java.security
securerandom.source=file:/dev/urandom
On Dec 18, 2015, at 10:41 AM, Constantine Yarovoy <ky...@gmail.com>> wrote:
Jonathan, please check my jstack dump during startup (which lasts at about 4-5 minutes):
[root@kk-ambari bin]# ./jps -l
16691 sun.tools.jps.Jps
16650 org.apache.ambari.server.controller.AmbariServer
[root@kk-ambari bin]# ./jstack -F -l 16650
Attaching to process ID 16650, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.40-b25
Deadlock Detection:
No deadlocks found.
Thread 16664: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Interpreted frame)
- com.google.inject.internal.util.$Finalizer.run() @bci=5, line=114 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 16659: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 16658: (state = BLOCKED)
Locked ownable synchronizers:
- None
Thread 16657: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Interpreted frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 16656: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=36, line=157 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 16651: (state = IN_NATIVE)
- java.io.FileInputStream.readBytes(byte[], int, int) @bci=0 (Interpreted frame)
- java.io.FileInputStream.read(byte[], int, int) @bci=4, line=255 (Interpreted frame)
- sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(byte[]) @bci=19, line=539 (Interpreted frame)
- sun.security.provider.SeedGenerator.generateSeed(byte[]) @bci=4, line=144 (Interpreted frame)
- sun.security.provider.SecureRandom$SeederHolder.<clinit>() @bci=20, line=203 (Interpreted frame)
- sun.security.provider.SecureRandom.engineNextBytes(byte[]) @bci=21, line=221 (Interpreted frame)
- java.security.SecureRandom.nextBytes(byte[]) @bci=5, line=468 (Interpreted frame)
- org.snmp4j.security.Salt.<init>() @bci=17, line=55 (Interpreted frame)
- org.snmp4j.security.Salt.getInstance() @bci=10, line=80 (Interpreted frame)
- org.snmp4j.security.PrivDES.<init>() @bci=5, line=57 (Interpreted frame)
- org.snmp4j.security.SecurityProtocols.addDefaultProtocols() @bci=362, line=152 (Interpreted frame)
- org.snmp4j.Snmp.initMessageDispatcher() @bci=61, line=225 (Interpreted frame)
- org.snmp4j.Snmp.<init>(org.snmp4j.TransportMapping) @bci=5, line=251 (Interpreted frame)
- org.apache.ambari.server.notifications.dispatchers.SNMPDispatcher.<init>() @bci=12, line=92 (Interpreted frame)
- sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=85, line=62 (Interpreted frame)
- sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Compiled frame)
- java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=422 (Compiled frame)
- java.lang.Class.newInstance() @bci=138, line=442 (Interpreted frame)
- org.apache.ambari.server.controller.ControllerModule.bindNotificationDispatchers() @bci=134, line=560 (Interpreted frame)
- org.apache.ambari.server.controller.ControllerModule.configure() @bci=543, line=341 (Interpreted frame)
- com.google.inject.AbstractModule.configure(com.google.inject.Binder) @bci=31, line=59 (Interpreted frame)
- com.google.inject.spi.Elements$RecordingBinder.install(com.google.inject.Module) @bci=31, line=223 (Interpreted frame)
- com.google.inject.spi.Elements.getElements(com.google.inject.Stage, java.lang.Iterable) @bci=40, line=101 (Interpreted frame)
- com.google.inject.internal.InjectorShell$Builder.build(com.google.inject.internal.Initializer, com.google.inject.internal.ProcessedBindingData, com.google.inject.internal.util.$Stopwatch, com.google.inject.internal.Errors) @bci=99, line=133 (Interpreted frame)
- com.google.inject.internal.InternalInjectorCreator.build() @bci=48, line=103 (Interpreted frame)
- com.google.inject.Guice.createInjector(com.google.inject.Stage, java.lang.Iterable) @bci=15, line=95 (Interpreted frame)
- com.google.inject.Guice.createInjector(java.lang.Iterable) @bci=4, line=72 (Interpreted frame)
- com.google.inject.Guice.createInjector(com.google.inject.Module[]) @bci=4, line=62 (Interpreted frame)
- org.apache.ambari.server.controller.AmbariServer.main(java.lang.String[]) @bci=14, line=706 (Interpreted frame)
Locked ownable synchronizers:
- None
As far as I understand, it gets stuck waiting for entropy to build up ?
I don't think it's a bug, because I'm using jdk 1.8.
Any workaround ? I'm launching Ambari server in Centos 7 os running on Openstack VM.
2015-11-16 20:58 GMT+02:00 Jonathan Hurley <jh...@hortonworks.com>>:
What this step is doing is loading classes which match an interface and binding them as individual alert dispatchers in Guice. I haven’t experienced any slowdown starting Ambari server - usually starts up in about 10 seconds total. Can you provide a jstack dump during your startup so we can see what the various threads are doing? I’m mainly concerned with the main Ambari thread that would be initializing this stuff.
> On Nov 16, 2015, at 3:52 AM, Constantine Yarovoy <ky...@gmail.com>> wrote:
>
> Hi all.
>
> I'm developing my own stack for Ambari and I often need to change master/slave component code in Python. And I have 2 questions regarding this:
>
> 1. What is the fastest way to make Ambari understand that stack has changed and to use updated code ? The only way it works for me now is to restart server (service ambari-server restart)
>
> Is it possible to do it any other way without restarting the server ?
>
> 2. Ambari server start procedure really takes too long. I'm using Centos 7 and after starting the service it takes at about 4-8 minutes for it to actually bind to port 8080 so that web ui becomes available. Tailing -f ambari-server.log I've notices that the biggest delay during start is on this step:
>
> 16 Nov 2015 08:47:13,392 INFO [main] ControllerModule:560 - Binding and registering notification dispatcher class org.apache.ambari.server.notifications.dispatchers.AlertScriptDispatcher
>
> Maybe someone experienced the same behavior and there is a way to speed up this step?
>
> Thanks in advance.
>
> --
> Kostiantyn Yarovyi
>
--
Regards,
Kostiantyn Yarovyi
CIO, ITIL Expert, Cisco CCNP, Microsoft MCITP
+38 (099) 148-96-28 | skype:kyarovoy