You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Attila Sasvari (JIRA)" <ji...@apache.org> on 2017/10/11 09:35:02 UTC
[jira] [Commented] (OOZIE-2775) Oozie server does not stop if there
is an exception during service initalization at startup
[ https://issues.apache.org/jira/browse/OOZIE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200015#comment-16200015 ]
Attila Sasvari commented on OOZIE-2775:
---------------------------------------
There is an easier way to reproduce it:
- start the Oozie server, and stop it. Change permission so that {{/tmp/embedded-jetty-jsp}} (or the location set by {{oozie.jsp.tmp.dir}}) is not writable by the user that starts Oozie.
- try to restart Oozie
{code}
2017-10-10 13:31:46,871 WARN org.eclipse.jetty.webapp.WebAppContext: SERVER[] Failed startup of context o.e.j.w.WebAppContext@42507640{/oozie,file:///usr/lib/oozie/embedded-oozie-server/webapp/,null}
java.lang.IllegalStateException: Failed to delete temp dir /tmp/embedded-jetty-jsp
at org.eclipse.jetty.webapp.WebInfConfiguration.configureTempDirectory(WebInfConfiguration.java:376)
at org.eclipse.jetty.webapp.WebInfConfiguration.resolveTempDirectory(WebInfConfiguration.java:275)
at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:69)
at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:485)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:521)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:422)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:389)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.oozie.server.EmbeddedOozieServer.start(EmbeddedOozieServer.java:189)
at org.apache.oozie.server.EmbeddedOozieServer.main(EmbeddedOozieServer.java:215)
{code}
Oozie services will start but the server itself will return 503. It might worth to file a new JIRA to handle this startup issue specifically, but it is up to you [~andras.piros]. What we could change:
- make an extra check at startup before starting the jetty server. If we see permission problems, we log a meaningful error message and fail fast.
- create & use /tmp/embedded_jetty_jsp_[OOZIE_PID] instead of /tmp/embedded_jetty_jsp. "Old" jetty jsp directories should be purged though.
> Oozie server does not stop if there is an exception during service initalization at startup
> -------------------------------------------------------------------------------------------
>
> Key: OOZIE-2775
> URL: https://issues.apache.org/jira/browse/OOZIE-2775
> Project: Oozie
> Issue Type: Bug
> Reporter: Attila Sasvari
> Assignee: Andras Piros
>
> OOZIE-2666 brought in embedded jetty in Oozie. {{com.google.inject.guice}} was selected for dependency management. Unfortunately if there is an exception at starting Oozie services (e.g. JPAService initialization, called via https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L194), Oozie won't stop if it cannot destroy Oozie services.
> In OOZIE-2666 a shutdown hook (https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java#L184) was added to print out startup errors and exceptions, destroy pending oozie services, and exit eventually, but it does not work.
> Steps to reproduce a startup failure (for example simulate that Oozie cannot find required database driver):
> {code}
> $ cd oozie
> $ mvn clean install assembly:single -DskipTests -DjavaVersion=1.8 -DtargetVersion=1.7 -Phadoop-2 -Dhadoop.version=2.6.0 -Puber;
> $ cd distro/target/oozie-4.4.0-SNAPSHOT-distro/oozie-4.4.0-SNAPSHOT/
> $ export JETTY_OPTS="-Djavax.net.debug=all -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y -Dlog4j.logger.org.eclipse.jetty=TRACE -Dlog4j.configuration=./conf/oozie-log4j.properties"
> $ bin/oozied.sh run
> $ mv embedded-oozie-server/webapp/WEB-INF/lib/derby-10.10.1.1.jar ..
> <attach to the process with a debugger and continue execution>
> {code}
> Exception is printed to stdout
> {code}
> <openjpa-2.4.1-r422266:1730418 fatal general error> org.apache.openjpa.persistence.PersistenceException: Cannot load JDBC driver class 'org.apache.derby.jdbc.EmbeddedDriver'
> at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:106)
> at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
> at org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1520)
> at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:533)
> at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:458)
> at org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:121)
> at org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
> at org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)
> at org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)
> at org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)
> at org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:642)
> at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:202)
> at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:154)
> at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:226)
> at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:153)
> at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:59)
> at org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:514)
> at org.apache.oozie.service.JPAService.init(JPAService.java:215)
> at org.apache.oozie.service.Services.setServiceInternal(Services.java:387)
> at org.apache.oozie.service.Services.setService(Services.java:373)
> at org.apache.oozie.service.Services.loadServices(Services.java:305)
> at org.apache.oozie.service.Services.init(Services.java:213)
> at org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:31)
> at org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:25)
> at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
> at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
> at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
> at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
> at org.apache.oozie.server.EmbeddedOozieServer.main(EmbeddedOozieServer.java:194)
> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver class 'org.apache.derby.jdbc.EmbeddedDriver'
> at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1429)
> at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371)
> at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
> at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
> at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:86)
> at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:90)
> ... 48 more
> Caused by: java.lang.ClassNotFoundException: org.apache.derby.jdbc.EmbeddedDriver
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1420)
> ... 53 more
> Exception in thread "main" java.lang.ExceptionInInitializerError: Could not instantiate Oozie services. [e.message=E0103: Could not load service classes, Cannot load JDBC driver class 'org.apache.derby.jdbc.EmbeddedDriver']
> at org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:36)
> at org.apache.oozie.server.guice.ServicesProvider.get(ServicesProvider.java:25)
> at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
> at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
> at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
> at com.google.inject.Scopes$1$1.get(Scopes.java:65)
> at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
> at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
> at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
> at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
> at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
> at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
> at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
> at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
> at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
> at org.apache.oozie.server.EmbeddedOozieServer.main(EmbeddedOozieServer.java:194)
> {code}
> I checked the JVM process with {{jstack}} and saw:
> {code}
> "JDWP Transport Listener: dt_socket" #45 daemon prio=10 os_prio=31 tid=0x00007fb70b001000 nid=0x3d0b runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "DestroyJavaVM" #44 prio=5 os_prio=31 tid=0x00007fb70ad9b000 nid=0x1003 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Attach Listener" #39 daemon prio=9 os_prio=31 tid=0x00007fb70b383000 nid=0x670f waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Scheduler-0" #29 prio=5 os_prio=31 tid=0x00007fb70da9b000 nid=0x5e0f waiting on condition [0x0000700001ae4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006c00187f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "FileWatchdog" #15 daemon prio=5 os_prio=31 tid=0x00007fb70b2a3000 nid=0x5a03 waiting on condition [0x00007000019e1000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
> "com.google.inject.internal.util.$Finalizer" #14 daemon prio=5 os_prio=31 tid=0x00007fb70abaf800 nid=0x5803 in Object.wait() [0x00007000018de000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000006c00188d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000006c00188d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
> "Service Thread" #12 daemon prio=9 os_prio=31 tid=0x00007fb70e834000 nid=0x5403 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C1 CompilerThread3" #11 daemon prio=9 os_prio=31 tid=0x00007fb70e00a800 nid=0x5203 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread2" #10 daemon prio=9 os_prio=31 tid=0x00007fb70a80f000 nid=0x5003 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" #9 daemon prio=9 os_prio=31 tid=0x00007fb70c002000 nid=0x4e03 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" #8 daemon prio=9 os_prio=31 tid=0x00007fb70d81a000 nid=0x1707 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "JDWP Event Helper Thread" #6 daemon prio=10 os_prio=31 tid=0x00007fb70a80e000 nid=0x4c03 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fb70b041000 nid=0x3e0f runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fb70e002000 nid=0x3803 in Object.wait() [0x0000700000d3a000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000006c00265d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000006c00265d8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fb70d802000 nid=0x3603 in Object.wait() [0x0000700000c37000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000006c0026670> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x00000006c0026670> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> "VM Thread" os_prio=31 tid=0x00007fb70b82b000 nid=0x3403 runnable
> "GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fb70d003800 nid=0x2403 runnable
> "GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fb70d004000 nid=0x2603 runnable
> "GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fb70d004800 nid=0x2803 runnable
> "GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fb70a80a800 nid=0x2a03 runnable
> "GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007fb70a80b000 nid=0x2c03 runnable
> "GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007fb70a80b800 nid=0x2e03 runnable
> "GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007fb70d005000 nid=0x3003 runnable
> "GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007fb70d006000 nid=0x3203 runnable
> "VM Periodic Task Thread" os_prio=31 tid=0x00007fb70b042000 nid=0x5603 waiting on condition
> JNI global references: 5741
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)