You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/03/01 22:54:00 UTC
[jira] [Commented] (NIFI-9741) Encountering "IOException: Stream is closed" when using Avro Writer
[ https://issues.apache.org/jira/browse/NIFI-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499798#comment-17499798 ]
ASF subversion and git services commented on NIFI-9741:
-------------------------------------------------------
Commit a327ba478ad87d28342c582de0b683b672fd4525 in nifi's branch refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a327ba4 ]
NIFI-9741: Make the close() method of WriteAvroResultWithExternalSchema idempotent
Signed-off-by: Matthew Burgess <ma...@apache.org>
This closes #5818
> Encountering "IOException: Stream is closed" when using Avro Writer
> -------------------------------------------------------------------
>
> Key: NIFI-9741
> URL: https://issues.apache.org/jira/browse/NIFI-9741
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Fix For: 1.16.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Avro Writer (WriteAvroResultWithExternalSchema) has a close method that is not idempotent. As a result, if the close() method is called multiple times, it can cause Exceptions to be thrown and potentially even cause data duplication or corruption, depending on the underlying OutputStream, as it could flush data to an OutputStream, and the OutputStream may block close() methods; a second call to WriteAvroResultWithExternalSchema.close() could then flush more data to the stream.
> This can produce a stack trace such as:
> {code:java}
> 2022-03-01 11:06:11,677 ERROR [Timer-Driven Process Thread-5] o.a.n.processors.standard.ValidateRecord ValidateRecord[id=39b23e8a-5b1a-110d-3bcc-5dca95722468] Failed to close Record Writer: java.io.IOException: Stream is closed
> ↳ causes: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to write to Content Repository for StandardFlowFileRecord[uuid=b57f14e0-2498-4e9d-8730-4b2e0c9dc874,claim=,offset=0,name=0076a439-5911-4178-b0e6-c7296c0f09ae,size=0]
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to write to Content Repository for StandardFlowFileRecord[uuid=b57f14e0-2498-4e9d-8730-4b2e0c9dc874,claim=,offset=0,name=0076a439-5911-4178-b0e6-c7296c0f09ae,size=0]
> at org.apache.nifi.controller.repository.StandardProcessSession$7.write(StandardProcessSession.java:2994)
> at org.apache.nifi.controller.repository.io.TaskTerminationOutputStream.write(TaskTerminationOutputStream.java:62)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at org.apache.avro.io.BufferedBinaryEncoder$OutputStreamSink.innerFlush(BufferedBinaryEncoder.java:220)
> at org.apache.avro.io.BufferedBinaryEncoder.flush(BufferedBinaryEncoder.java:85)
> at org.apache.avro.io.BlockingBinaryEncoder.flush(BlockingBinaryEncoder.java:251)
> at org.apache.nifi.avro.WriteAvroResultWithExternalSchema.flush(WriteAvroResultWithExternalSchema.java:95)
> at org.apache.nifi.avro.WriteAvroResultWithExternalSchema.close(WriteAvroResultWithExternalSchema.java:112)
> at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
> at com.sun.proxy.$Proxy193.close(Unknown Source)
> at org.apache.nifi.processors.standard.ValidateRecord.closeQuietly(ValidateRecord.java:459)
> at org.apache.nifi.processors.standard.ValidateRecord.onTrigger(ValidateRecord.java:434)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1284)
> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Stream is closed
> at org.apache.nifi.controller.repository.io.DisableOnCloseOutputStream.write(DisableOnCloseOutputStream.java:46)
> at org.apache.nifi.stream.io.ByteCountingOutputStream.write(ByteCountingOutputStream.java:49)
> at org.apache.nifi.controller.repository.StandardProcessSession$7.write(StandardProcessSession.java:2989)
> ... 29 common frames omitted {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)