You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by sohimankotia <so...@gmail.com> on 2019/02/24 03:14:54 UTC

Re: Flink Streaming Job Task is getting cancelled silently and causing job to restart

Hi Erik,

Are your suggesting all options together ?

Which of version of flink has this solved ? I am currently using 1.5.5 .

-Thanks
Sohi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink Streaming Job Task is getting cancelled silently and causing job to restart

Posted by sohimankotia <so...@gmail.com>.

Thanks Andrey . 

Yeah will upgrade and see if same gets reproduced .


-Sohi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink Streaming Job Task is getting cancelled silently and causing job to restart

Posted by sohimankotia <so...@gmail.com>.

Hi Andrey,

I am using AvroSinkWriter (with Bucketing Sink) with compression enabled .
Looks like StreamingFileSink does not have direct support for
AvroSinkWriter.  Sequence File Format is there for StreamingFileSink , but
looks like it roll files on every checkpoint (OnCheckpointRollingPolicy)
which can create lots of small files as we have checkpointing enabled every
2 minutes .



If following issue is still there for 1.7.2 version:

(d9d0f719652f87dfb2ec663b46ef2e47) switched from RUNNING to FAILED.
TimerException{java.nio.channels.ClosedByInterruptException}
	at
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:288)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedByInterruptException
	at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:478)
	at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
	at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:618)


Thanks
Sohi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink Streaming Job Task is getting cancelled silently and causing job to restart

Posted by Andrey Zagrebin <an...@ververica.com>.

Hi Sohi,

I would also recommend trying the newer StreamingFileSink which is
available in Flink 1.7.x [1].

Best,
Andrey

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/streamfile_sink.html

On Sun, Feb 24, 2019 at 4:14 AM sohimankotia <so...@gmail.com> wrote:

> Hi Erik,
>
> Are your suggesting all options together ?
>
> Which of version of flink has this solved ? I am currently using 1.5.5 .
>
> -Thanks
> Sohi
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>