You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Christian Gumpert (Jira)" <ji...@apache.org> on 2021/01/07 12:46:00 UTC
[jira] [Created] (NIFI-8119) ExecuteSQL does not properly free
database ressources
Christian Gumpert created NIFI-8119:
---------------------------------------
Summary: ExecuteSQL does not properly free database ressources
Key: NIFI-8119
URL: https://issues.apache.org/jira/browse/NIFI-8119
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.11.2
Reporter: Christian Gumpert
We are using Nifi to ingest data from a Teradata database into our S3-based data lake using a typical pattern of GenerateTableFetch and ExecuteSQL processors. Our Teradata database tables contain columns of type CLOB (which contains some JSON data).
We have installed the Teradata JDBC driver from the Teradata Tools and Utilities package version 16.10.26.00 as described in this [Cloudera community article|https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783].
After having configured a DBConnectionPool service with the Teradata connection parameters we are able to execute our flow. The GenerateTableFetch processors generates flowfiles containing SQL Queries which are then executed by the ExecuteSQL processor.
After having processed the first 15 flowfiles the ExecuteSQL processor yields the following error:
{noformat}
2020-12-17T12:53:17+01:00 L921000109090A nifi-app.log: 2020-12-17 12:53:11,578 ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=afa23b0f-2e57-1fb6-d047-13646de03ebf] Unable to execute SQL select query call devezv_replworkedec.get_edec_meldung(561, 562, 2, 0); for StandardFlowFileRecord[uuid=ff7219a7-14e9-404e-a57a-28121653fed8,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1607610786368-4986, container=repo0, section=890], offset=701888, length=1077672],offset=32266,name=ff7219a7-14e9-404e-a57a-28121653fed8,size=58] due to java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.; routing to failure: java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:309)
at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:103)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:311)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:200)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:137)
at com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:331)
at com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecute(TDPreparedStatement.java:177)
at com.teradata.jdbc.jdbc_4.TDPreparedStatement.execute(TDPreparedStatement.java:2778)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
at org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:266)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){noformat}
After googling for a while it appears that the important part is
*java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.*
The [Teradata documentation for "Working with LOBs"|https://teradata-docs.s3.amazonaws.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_5.html#CHDGCHBB] contains information on how to properly free resources to avoid this limit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)