You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Shiv (Jira)" <ji...@apache.org> on 2022/10/17 12:36:00 UTC
[jira] [Updated] (NIFI-10657) When multiple CLOB columns in a table are configured for processing in Oracle, NIFI is throwing Out of Memory exception.
[ https://issues.apache.org/jira/browse/NIFI-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shiv updated NIFI-10657:
------------------------
Description:
This issue is happening because Nifi Processor PutDatabaseRecord is not clearing CLOBs once the data is written to the databases. The Nifi Processor code needs to be modified to clear CLOB before closing Resultsets.
Table being processed has four CLOB data type columns. Average row size is 10K, and each CLOB column average length is 2K
I don't have the permission to create a branch.. The below code change is needed to fix the issue *(tested against rel/nifi-1.17.0 branch)*
{code:java}
diff --git a/nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java b/nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java
index b78408c912..bfc5328603 100644
--- a/nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java
+++ b/nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java
@@ -280,6 +280,11 @@ public class JdbcCommon {
}
}
rec.put(i - 1, sb.toString());
+ try {
+ clob.free();
+ } catch (SQLFeatureNotSupportedException sfnse) {
+ // The driver doesn't support free, but allow processing to continue
+ }
} else {
rec.put(i - 1, null);
} {code}
was:
This issue is happening because Nifi Processor PutDatabaseRecord is not clearing CLOBs once the data is written to the databases.
The Nifi Processor code needs to be modified to clear CLOB before closing Resultsets.
Even though the issue is reproduced and tested for Oracle database, it could be a problem for other databases too.
> When multiple CLOB columns in a table are configured for processing in Oracle, NIFI is throwing Out of Memory exception.
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-10657
> URL: https://issues.apache.org/jira/browse/NIFI-10657
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.17.0
> Environment: Docker container
> Reporter: Shiv
> Priority: Major
> Labels: performance, pull-request-available
>
> This issue is happening because Nifi Processor PutDatabaseRecord is not clearing CLOBs once the data is written to the databases. The Nifi Processor code needs to be modified to clear CLOB before closing Resultsets.
> Table being processed has four CLOB data type columns. Average row size is 10K, and each CLOB column average length is 2K
> I don't have the permission to create a branch.. The below code change is needed to fix the issue *(tested against rel/nifi-1.17.0 branch)*
> {code:java}
> diff --git a/nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java b/nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java
> index b78408c912..bfc5328603 100644
> --- a/nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java
> +++ b/nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java
> @@ -280,6 +280,11 @@ public class JdbcCommon {
> }
> }
> rec.put(i - 1, sb.toString());
> + try {
> + clob.free();
> + } catch (SQLFeatureNotSupportedException sfnse) {
> + // The driver doesn't support free, but allow processing to continue
> + }
> } else {
> rec.put(i - 1, null);
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)