You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Mona Chitnis <ch...@yahoo-inc.com> on 2013/07/01 19:16:37 UTC
Re: Oozie org.apache.openjpa.lib.jdbc.ReportingSQLException ~once a
week
Also try increasing the Oozie configuration value of
oozie.service.JPAService.pool.max.active.conn to something higher, since
this seems to be hitting the issue of concurrent database operations not
able to execute. Depending on size of staging/production environment, this
value can go from 220 to 370 ish.
--
Mona
On 6/28/13 1:28 AM, "Harsh J" <ha...@cloudera.com> wrote:
>Hi Christian,
>
>What version of Oozie are you using here? And what DB is it running
>against?
>
>On Mon, Jun 17, 2013 at 4:18 PM, Christian Schneider
><cs...@gmail.com> wrote:
>> Hi, our Oozie instance (managed by Cloudera Manager) get a
>> org.apache.openjpa.lib.jdbc.ReportingSQLException almost once a week.
>>
>> When this happens, we see this Stacktraces in the log files:
>>
>> 2013-06-17 09:01:20,047 ERROR org.apache.oozie.command.wf.JobsXCommand:
>> USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Error,
>> 2013-06-17 09:01:53,330 WARN org.apache.oozie.service.JPAService:
>>USER[-]
>> GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] JPAExecutor
>> [WorkflowsJobGetJPAExecutor] ended with an active transaction, rolling
>>back
>> 2013-06-17 09:01:53,331 ERROR org.apache.oozie.command.wf.JobsXCommand:
>> USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] XException,
>> org.apache.oozie.command.CommandException: E0603: SQL error in
>>operation,
>> Cannot issue commit in a nested connection when there is a pending
>> operation in the parent connection. {prepstmnt 320277519 SELECT t0.id,
>> t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name,
>> t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time,
>> t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY
>> t0.created_time DESC [params=?]} [code=30000, state=X0Y66]
>> at org.apache.oozie.command.wf.JobsXCommand.execute(
>> JobsXCommand.java:72)
>> at org.apache.oozie.command.wf.JobsXCommand.execute(
>> JobsXCommand.java:32)
>> at org.apache.oozie.command.XCommand.call(XCommand.java:277)
>> at org.apache.oozie.DagEngine.getJobs(DagEngine.java:439)
>> at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs(
>> V1JobsServlet.java:314)
>> at org.apache.oozie.servlet.V1JobsServlet.getJobs(
>> V1JobsServlet.java:141)
>> at org.apache.oozie.servlet.BaseJobsServlet.doGet(
>> BaseJobsServlet.java:121)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>> at org.apache.oozie.servlet.JsonRestServlet.service(
>> JsonRestServlet.java:286)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>> at
>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>> ApplicationFilterChain.java:290)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
>> ApplicationFilterChain.java:206)
>> at org.apache.oozie.servlet.AuthFilter$2.doFilter(
>> AuthFilter.java:126)
>> at org.apache.hadoop.security.authentication.server.
>> AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
>> at
>>org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
>> at
>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>> ApplicationFilterChain.java:235)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
>> ApplicationFilterChain.java:206)
>> at org.apache.oozie.servlet.HostnameFilter.doFilter(
>> HostnameFilter.java:84)
>> at
>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>> ApplicationFilterChain.java:235)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
>> ApplicationFilterChain.java:206)
>> at org.apache.catalina.core.StandardWrapperValve.invoke(
>> StandardWrapperValve.java:233)
>> at org.apache.catalina.core.StandardContextValve.invoke(
>> StandardContextValve.java:191)
>> at org.apache.catalina.core.StandardHostValve.invoke(
>> StandardHostValve.java:127)
>> at org.apache.catalina.valves.ErrorReportValve.invoke(
>> ErrorReportValve.java:102)
>> at org.apache.catalina.core.StandardEngineValve.invoke(
>> StandardEngineValve.java:109)
>> at org.apache.catalina.connector.CoyoteAdapter.service(
>> CoyoteAdapter.java:293)
>> at org.apache.coyote.http11.Http11Processor.process(
>> Http11Processor.java:859)
>> at
>>org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.
>> process(Http11Protocol.java:602)
>> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(
>> JIoEndpoint.java:489)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: <openjpa-2.1.0-r422266:1071316 fatal general error>
>> org.apache.openjpa.persistence.PersistenceException: Cannot issue
>>commit in
>> a nested connection when there is a pending operation in the parent
>> connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status,
>> t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time,
>> t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE
>> t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000,
>> state=X0Y66]
>> at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(
>> DBDictionary.java:4869)
>> at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(
>> DBDictionary.java:4829)
>> at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
>> SQLExceptions.java:136)
>> at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
>> SQLExceptions.java:110)
>> at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
>> SQLExceptions.java:62)
>> at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$
>> PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:139)
>> at
>>org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005)
>> at
>>org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
>> at
>>org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
>> at org.apache.openjpa.kernel.DelegatingQuery.execute(
>> DelegatingQuery.java:542)
>> at org.apache.openjpa.persistence.QueryImpl.execute(
>> QueryImpl.java:305)
>> at org.apache.openjpa.persistence.QueryImpl.
>> getResultList(QueryImpl.java:319)
>> at
>>org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute(
>> WorkflowsJobGetJPAExecutor.java:247)
>> at
>>org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute(
>> WorkflowsJobGetJPAExecutor.java:40)
>> at
>>org.apache.oozie.service.JPAService.execute(JPAService.java:211)
>> at org.apache.oozie.command.wf.JobsXCommand.execute(
>> JobsXCommand.java:61)
>> ... 29 more
>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Cannot
>>issue
>> commit in a nested connection when there is a pending operation in the
>> parent connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name,
>> t0.status, t0.run, t0.user_name, t0.group_name, t0.created_time,
>> t0.start_time, t0.last_modified_time, t0.end_time, t0.external_id FROM
>> WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY t0.created_time DESC
>>[params=?]}
>> [code=30000, state=X0Y66]
>> at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(
>> LoggingConnectionDecorator.java:281)
>> at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(
>> LoggingConnectionDecorator.java:265)
>> at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.
>> access$700(LoggingConnectionDecorator.java:72)
>> at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$
>> LoggingConnection$LoggingPreparedStatement.executeQuery(
>> LoggingConnectionDecorator.java:1183)
>> at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.
>> executeQuery(DelegatingPreparedStatement.java:284)
>> at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$
>> CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1785)
>> at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.
>> executeQuery(DelegatingPreparedStatement.java:274)
>> at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$
>> PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:118)
>> ... 39 more
>> 2013-06-17 09:01:54,275 WARN org.apache.oozie.servlet.V1JobsServlet:
>> USER[hue] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[GET
>>
>>http://ip-10-0-1-149.eu-west-1.compute.internal:11000/oozie/v1/jobs?len=1
>>00&doAs=hue&filter=&user.name=hue&timezone=America%2FLos_Angeles&jobtype=
>>wf]
>> error[E0603], E0603: SQL error in operation, Cannot issue commit in a
>> nested connection when there is a pending operation in the parent
>> connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status,
>> t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time,
>> t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE
>> t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000,
>> state=X0Y66]
>> org.apache.oozie.servlet.XServletException: E0603: SQL error in
>>operation,
>> Cannot issue commit in a nested connection when there is a pending
>> operation in the parent connection. {prepstmnt 320277519 SELECTt0.id,
>> t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name,
>> t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time,
>> t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY
>> t0.created_time DESC [params=?]} [code=30000, state=X0Y66]
>> at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs(
>> V1JobsServlet.java:323)
>> at org.apache.oozie.servlet.V1JobsServlet.getJobs(
>> V1JobsServlet.java:141)
>> at org.apache.oozie.servlet.BaseJobsServlet.doGet(
>> BaseJobsServlet.java:121)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>> at org.apache.oozie.servlet.JsonRestServlet.service(
>> JsonRestServlet.java:286)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>> at
>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>> ApplicationFilterChain.java:290)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
>> ApplicationFilterChain.java:206)
>> at org.apache.oozie.servlet.AuthFilter$2.doFilter(
>> AuthFilter.java:126)
>> at org.apache.hadoop.security.authentication.server.
>> AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
>> at
>>org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
>> at
>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>> ApplicationFilterChain.java:235)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
>> ApplicationFilterChain.java:206)
>> at org.apache.oozie.servlet.HostnameFilter.doFilter(
>> HostnameFilter.java:84)
>> at
>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>> ApplicationFilterChain.java:235)
>> at org.apache.catalina.core.ApplicationFilterChain.doFilter(
>> ApplicationFilterChain.java:206)
>> at org.apache.catalina.core.StandardWrapperValve.invoke(
>> StandardWrapperValve.java:233)
>> at org.apache.catalina.core.StandardContextValve.invoke(
>> StandardContextValve.java:191)
>> at org.apache.catalina.core.StandardHostValve.invoke(
>> StandardHostValve.java:127)
>> at org.apache.catalina.valves.ErrorReportValve.invoke(
>> ErrorReportValve.java:102)
>> at org.apache.catalina.core.StandardEngineValve.invoke(
>> StandardEngineValve.java:109)
>> at org.apache.catalina.connector.CoyoteAdapter.service(
>> CoyoteAdapter.java:293)
>> at org.apache.coyote.http11.Http11Processor.process(
>> Http11Processor.java:859)
>> at
>>org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.
>> process(Http11Protocol.java:602)
>> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(
>> JIoEndpoint.java:489)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: org.apache.oozie.DagEngineException: E0603: SQL error in
>> operation, Cannot issue commit in a nested connection when there is a
>> pending operation in the parent connection. {prepstmnt 320277519 SELECT
>> t0.id, t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name,
>> t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time,
>> t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY
>> t0.created_time DESC [params=?]} [code=30000, state=X0Y66]
>> at org.apache.oozie.DagEngine.getJobs(DagEngine.java:442)
>> at org.apache.oozie.servlet.V1JobsServlet.getWorkflowJobs(
>> V1JobsServlet.java:314)
>> ... 25 more
>> Caused by: org.apache.oozie.command.CommandException: E0603: SQL error
>>in
>> operation, Cannot issue commit in a nested connection when there is a
>> pending operation in the parent connection. {prepstmnt 320277519 SELECT
>> t0.id, t0.app_name, t0.status, t0.run, t0.user_name, t0.group_name,
>> t0.created_time, t0.start_time, t0.last_modified_time, t0.end_time,
>> t0.external_id FROM WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY
>> t0.created_time DESC [params=?]} [code=30000, state=X0Y66]
>> at org.apache.oozie.command.wf.JobsXCommand.execute(
>> JobsXCommand.java:72)
>> at org.apache.oozie.command.wf.JobsXCommand.execute(
>> JobsXCommand.java:32)
>> at org.apache.oozie.command.XCommand.call(XCommand.java:277)
>> at org.apache.oozie.DagEngine.getJobs(DagEngine.java:439)
>> ... 26 more
>> Caused by: <openjpa-2.1.0-r422266:1071316 fatal general error>
>> org.apache.openjpa.persistence.PersistenceException: Cannot issue
>>commit in
>> a nested connection when there is a pending operation in the parent
>> connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name, t0.status,
>> t0.run, t0.user_name, t0.group_name, t0.created_time, t0.start_time,
>> t0.last_modified_time, t0.end_time, t0.external_id FROM WF_JOBS t0 WHERE
>> t0.bean_type = ? ORDER BY t0.created_time DESC [params=?]} [code=30000,
>> state=X0Y66]
>> at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(
>> DBDictionary.java:4869)
>> at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(
>> DBDictionary.java:4829)
>> at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
>> SQLExceptions.java:136)
>> at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
>> SQLExceptions.java:110)
>> at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(
>> SQLExceptions.java:62)
>> at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$
>> PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:139)
>> at
>>org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005)
>> at
>>org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
>> at
>>org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
>> at org.apache.openjpa.kernel.DelegatingQuery.execute(
>> DelegatingQuery.java:542)
>> at org.apache.openjpa.persistence.QueryImpl.execute(
>> QueryImpl.java:305)
>> at org.apache.openjpa.persistence.QueryImpl.
>> getResultList(QueryImpl.java:319)
>> at
>>org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute(
>> WorkflowsJobGetJPAExecutor.java:247)
>> at
>>org.apache.oozie.executor.jpa.WorkflowsJobGetJPAExecutor.execute(
>> WorkflowsJobGetJPAExecutor.java:40)
>> at
>>org.apache.oozie.service.JPAService.execute(JPAService.java:211)
>> at org.apache.oozie.command.wf.JobsXCommand.execute(
>> JobsXCommand.java:61)
>> ... 29 more
>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Cannot
>>issue
>> commit in a nested connection when there is a pending operation in the
>> parent connection. {prepstmnt 320277519 SELECT t0.id, t0.app_name,
>> t0.status, t0.run, t0.user_name, t0.group_name, t0.created_time,
>> t0.start_time, t0.last_modified_time, t0.end_time, t0.external_id FROM
>> WF_JOBS t0 WHERE t0.bean_type = ? ORDER BY t0.created_time DESC
>>[params=?]}
>> [code=30000, state=X0Y66]
>> at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(
>> LoggingConnectionDecorator.java:281)
>> at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(
>> LoggingConnectionDecorator.java:265)
>> at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.
>> access$700(LoggingConnectionDecorator.java:72)
>> at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$
>> LoggingConnection$LoggingPreparedStatement.executeQuery(
>> LoggingConnectionDecorator.java:1183)
>> at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.
>> executeQuery(DelegatingPreparedStatement.java:284)
>> at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$
>> CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1785)
>> at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.
>> executeQuery(DelegatingPreparedStatement.java:274)
>> at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$
>> PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:118)
>> ... 39 more
>>
>> After resting the service, the systems will run fine.
>>
>> I have no idea where to debug or how to solve this.
>>
>>
>> Best Regards,
>> Christian.
>
>
>
>--
>Harsh J