You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Jonathan Keller (Jira)" <ji...@apache.org> on 2021/02/23 21:46:00 UTC
[jira] [Updated] (NIFI-8254) Timestamp and enums in Avro schema not
working in PutDatabaseRecord on Oracle
[ https://issues.apache.org/jira/browse/NIFI-8254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Keller updated NIFI-8254:
----------------------------------
Description:
There appears to be a regression in the PutDatabaseRecord processor with 1.13.0 (works in 1.12.1)
I have an Avro schema for a record which contained enum fields and timestamps. After upgrading, a processor which had been working was failing on errors on both of these field types. One stated it could not find the record type, the other with the exception and stack trace below. The fields in question were defined as below.
The files which were erroring out were the exact same files, and no changes had been made to the schema or the CSV reader controller between the tests. I was also able to successfully move the flow.xml.gz file back to the older version of NiFi and the PutDatabaseRecord processor was able to work again to insert the database records.
{noformat}
{
"name": "PER_ORG",
"type": {
"type": "enum",
"name": "PerOrgFlag",
"symbols": [
"EMP",
"CWR"
]
}
}, {
"name": "UPD_BT_DTM",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
},
{noformat}
{noformat}
2021-02-23 11:44:26,496 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.PutDatabaseRecord PutDatabaseRecord[id=93b478c8-0177-1000-1ca3-59b4189b1ecb] Failed to put Records to database for StandardFlowFileRecord[uuid=fd4e3be1-8f7b-4a7a-bf45-2e98627f732a,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1614107687354-5, container=default, section=5], offset=682549, length=6820],offset=0,name=DDODS_DVCMP_PS_JOB_2021-02-09_021816_2648.dat,size=6820]. Routing to failure.: java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected NUMBER got TIMESTAMP
java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected NUMBER got TIMESTAMP
at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9711)
at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1447)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9487)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:237)
at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at sun.reflect.GeneratedMethodAccessor583.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
at com.sun.proxy.$Proxy360.executeBatch(Unknown Source)
at org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:751)
at org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
at org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
eThere appears to be a regression in the PutDatabaseRecord processor with 1.13.0 (works in 1.12.1)
I have an Avro schema for a record which contained enum fields and timestamps. After upgrading, a processor which had been working was failing on errors on both of these field types. One stated it could not find the record type, the other with the exception and stack trace below. The fields in question were defined as below. The files which were erroring out were the exact same files, and no changes had been made to the schema or the CSV reader controller between the tests. I was also able to successfully move the flow.xml.gz file back to the older version of NiFi and the PutDatabaseRecord processor was able to work again to insert the database records.
{noformat}
{
"name": "PER_ORG",
"type": {
"type": "enum",
"name": "PerOrgFlag",
"symbols": [
"EMP",
"CWR"
]
}
}, {
"name": "UPD_BT_DTM",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
},
{noformat}
{noformat}
2021-02-23 11:44:26,496 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.PutDatabaseRecord PutDatabaseRecord[id=93b478c8-0177-1000-1ca3-59b4189b1ecb] Failed to put Records to database for StandardFlowFileRecord[uuid=fd4e3be1-8f7b-4a7a-bf45-2e98627f732a,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1614107687354-5, container=default, section=5], offset=682549, length=6820],offset=0,name=DDODS_DVCMP_PS_JOB_2021-02-09_021816_2648.dat,size=6820]. Routing to failure.: java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected NUMBER got TIMESTAMP
java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected NUMBER got TIMESTAMP
at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9711)
at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1447)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9487)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:237)
at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at sun.reflect.GeneratedMethodAccessor583.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
at com.sun.proxy.$Proxy360.executeBatch(Unknown Source)
at org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:751)
at org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
at org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> Timestamp and enums in Avro schema not working in PutDatabaseRecord on Oracle
> -----------------------------------------------------------------------------
>
> Key: NIFI-8254
> URL: https://issues.apache.org/jira/browse/NIFI-8254
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.13.0
> Reporter: Jonathan Keller
> Priority: Major
>
> There appears to be a regression in the PutDatabaseRecord processor with 1.13.0 (works in 1.12.1)
> I have an Avro schema for a record which contained enum fields and timestamps. After upgrading, a processor which had been working was failing on errors on both of these field types. One stated it could not find the record type, the other with the exception and stack trace below. The fields in question were defined as below.
> The files which were erroring out were the exact same files, and no changes had been made to the schema or the CSV reader controller between the tests. I was also able to successfully move the flow.xml.gz file back to the older version of NiFi and the PutDatabaseRecord processor was able to work again to insert the database records.
> {noformat}
> {
> "name": "PER_ORG",
> "type": {
> "type": "enum",
> "name": "PerOrgFlag",
> "symbols": [
> "EMP",
> "CWR"
> ]
> }
> }, {
> "name": "UPD_BT_DTM",
> "type": {
> "type": "long",
> "logicalType": "timestamp-millis"
> }
> },
> {noformat}
>
> {noformat}
> 2021-02-23 11:44:26,496 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.PutDatabaseRecord PutDatabaseRecord[id=93b478c8-0177-1000-1ca3-59b4189b1ecb] Failed to put Records to database for StandardFlowFileRecord[uuid=fd4e3be1-8f7b-4a7a-bf45-2e98627f732a,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1614107687354-5, container=default, section=5], offset=682549, length=6820],offset=0,name=DDODS_DVCMP_PS_JOB_2021-02-09_021816_2648.dat,size=6820]. Routing to failure.: java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected NUMBER got TIMESTAMP
> java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected NUMBER got TIMESTAMP
> at oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9711)
> at oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1447)
> at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9487)
> at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:237)
> at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
> at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
> at sun.reflect.GeneratedMethodAccessor583.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
> at com.sun.proxy.$Proxy360.executeBatch(Unknown Source)
> at org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:751)
> at org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
> at org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)