You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by ka...@verizon.com on 2016/12/26 19:52:04 UTC
RE: [E] writeBufferLowWaterMark cannot be greater than
writeBufferHighWaterMark error
Hi,
I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error frequently ... and those task managers not processing messages after that error.
What could be wrong in my configuration? What I need to do to avoid this error.?
Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072
2016-12-26 14:26:06,548| |WARN io.netty.bootstrap.ServerBootstrap - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /138.83.31.9:41304]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073
at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Thanks
Kanagaraj Vengidasamy
RTCI
7701 E Telecom PKWY
Temple Terrace, FL 33637
O 813.978.4372 | M 813.455.9757
RE: [E] writeBufferLowWaterMark cannot be greater than
writeBufferHighWaterMark error
Posted by ka...@verizon.com.
Hi Fabian,
In my case, Since it is stream processing, after that error , that task manager stuck and not taking any new messages.
Can you let me know how many task managers can I run in 8 core x 32GB machine.? I am using Heap size as 4096 for each task manager.
Thanks
[Verizon]<http://www.verizon.com/>
Kanagaraj Vengidasamy
RTCI
7701 E Telecom PKWY
Temple Terrace, FL 33637
O 813.978.4372 | M 813.455.9757
<http://www.facebook.com/verizon> [Twitter] <http://twitter.com/verizon> [LinkedIn] <http://www.linkedin.com/company/verizon>
From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: Tuesday, December 27, 2016 2:53 PM
To: user@flink.apache.org
Cc: Ufuk Celebi
Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error
Hi,
I reproduced the issue with Flink 1.1.4 and the 1.2.0 release branch.
The WARN log statement and the IllegalArgumentException are thrown by Netty.
Not sure what the implications are. My batch jobs finished successfully, so maybe the bad configuration options are just ignored.
Would be good to check this though.
Thanks,
Fabian
2016-12-27 16:10 GMT+01:00 <ka...@verizon.com>>:
Thanks Fabian,
For default 65536 also got the same error . That is the reason increased to see it is helping or not.
Thanks
[Verizon]<http://www.verizon.com/>
Kanagaraj Vengidasamy
RTCI
7701 E Telecom PKWY
Temple Terrace, FL 33637
O 813.978.4372<tel:(813)%20978-4372> | M 813.455.9757<tel:(813)%20455-9757>
<http://www.facebook.com/verizon> [Twitter] <http://twitter.com/verizon> [LinkedIn] <http://www.linkedin.com/company/verizon>
From: Fabian Hueske [mailto:fhueske@gmail.com<ma...@gmail.com>]
Sent: Tuesday, December 27, 2016 10:01 AM
To: user@flink.apache.org<ma...@flink.apache.org>
Cc: Ufuk Celebi
Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error
Hi Kanagaraj,
I would assume that the issue is caused by this configuration parameter:
taskmanager.memory.segment-size: 131072
I think the maximum possible value given Netty's "writeBufferHighWaterMark" parameter is 65536.
There might be a way to tune Netty's parameters but I don't know how to do that. Maybe Ufuk (in CC) knows better.
Is there any particular reason why want to increase the network buffer size instead of keeping the default?
Did you find that the default parameter gives insufficient performance?
Best, Fabian
2016-12-26 20:52 GMT+01:00 <ka...@verizon.com>>:
Hi,
I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error frequently ... and those task managers not processing messages after that error.
What could be wrong in my configuration? What I need to do to avoid this error.?
Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072
2016-12-26 14:26:06,548| |WARN io.netty.bootstrap.ServerBootstrap - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812<http://138.83.31.4:60812> => /138.83.31.9:41304<http://138.83.31.9:41304>]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073
at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Thanks
Kanagaraj Vengidasamy
RTCI
7701 E Telecom PKWY
Temple Terrace, FL 33637
O 813.978.4372<tel:(813)%20978-4372> | M 813.455.9757<tel:(813)%20455-9757>
Re: [E] writeBufferLowWaterMark cannot be greater than
writeBufferHighWaterMark error
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
I reproduced the issue with Flink 1.1.4 and the 1.2.0 release branch.
The WARN log statement and the IllegalArgumentException are thrown by Netty.
Not sure what the implications are. My batch jobs finished successfully, so
maybe the bad configuration options are just ignored.
Would be good to check this though.
Thanks,
Fabian
2016-12-27 16:10 GMT+01:00 <ka...@verizon.com>:
> Thanks Fabian,
>
>
>
> For default 65536 also got the same error . That is the
> reason increased to see it is helping or not.
>
>
>
> Thanks
>
>
> [image: Verizon] <http://www.verizon.com/>
>
> Kanagaraj Vengidasamy
> RTCI
>
> 7701 E Telecom PKWY
> Temple Terrace, FL 33637
>
> O 813.978.4372 <(813)%20978-4372> | M 813.455.9757 <(813)%20455-9757>
>
> [image: Facebook] <http://www.facebook.com/verizon> [image: Twitter]
> <http://twitter.com/verizon> [image: LinkedIn]
> <http://www.linkedin.com/company/verizon>
>
>
>
> *From:* Fabian Hueske [mailto:fhueske@gmail.com]
> *Sent:* Tuesday, December 27, 2016 10:01 AM
> *To:* user@flink.apache.org
> *Cc:* Ufuk Celebi
> *Subject:* Re: [E] writeBufferLowWaterMark cannot be greater than
> writeBufferHighWaterMark error
>
>
>
> Hi Kanagaraj,
>
> I would assume that the issue is caused by this configuration parameter:
>
> taskmanager.memory.segment-size: 131072
>
> I think the maximum possible value given Netty's
> "writeBufferHighWaterMark" parameter is 65536.
>
> There might be a way to tune Netty's parameters but I don't know how to do
> that. Maybe Ufuk (in CC) knows better.
>
>
> Is there any particular reason why want to increase the network buffer
> size instead of keeping the default?
>
> Did you find that the default parameter gives insufficient performance?
>
> Best, Fabian
>
>
>
> 2016-12-26 20:52 GMT+01:00 <ka...@verizon.com>:
>
>
> Hi,
>
> I am getting writeBufferLowWaterMark cannot be greater than
> writeBufferHighWaterMark error frequently ... and those task managers not
> processing messages after that error.
> What could be wrong in my configuration? What I need to do to avoid this
> error.?
>
> Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8
> slots)
> taskmanager.heap.mb: 4096
> taskmanager.numberOfTaskSlots: 8
> taskmanager.network.numberOfBuffers: 22000
> taskmanager.memory.segment-size: 131072
>
>
> 2016-12-26 14:26:06,548| |WARN io.netty.bootstrap.ServerBootstrap -
> Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /
> 138.83.31.9:41304]
> java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be
> greater than writeBufferHighWaterMark (65536): 131073
> at io.netty.channel.DefaultChannelConfig.
> setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
> at io.netty.channel.socket.DefaultSocketChannelConfig.
> setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
> at io.netty.channel.socket.DefaultSocketChannelConfig.
> setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
> at io.netty.channel.DefaultChannelConfig.setOption(
> DefaultChannelConfig.java:183)
> at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(
> DefaultSocketChannelConfig.java:121)
> at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.
> channelRead(ServerBootstrap.java:238)
> at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:324)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(
> DefaultChannelPipeline.java:847)
> at io.netty.channel.nio.AbstractNioMessageChannel$
> NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
> NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
> NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
> NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks
> Kanagaraj Vengidasamy
> RTCI
>
> 7701 E Telecom PKWY
> Temple Terrace, FL 33637
>
> O 813.978.4372 <(813)%20978-4372> | M 813.455.9757 <(813)%20455-9757>
>
>
>
>
>
RE: [E] writeBufferLowWaterMark cannot be greater than
writeBufferHighWaterMark error
Posted by ka...@verizon.com.
Thanks Fabian,
For default 65536 also got the same error . That is the reason increased to see it is helping or not.
Thanks
[Verizon]<http://www.verizon.com/>
Kanagaraj Vengidasamy
RTCI
7701 E Telecom PKWY
Temple Terrace, FL 33637
O 813.978.4372 | M 813.455.9757
<http://www.facebook.com/verizon> [Twitter] <http://twitter.com/verizon> [LinkedIn] <http://www.linkedin.com/company/verizon>
From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: Tuesday, December 27, 2016 10:01 AM
To: user@flink.apache.org
Cc: Ufuk Celebi
Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error
Hi Kanagaraj,
I would assume that the issue is caused by this configuration parameter:
taskmanager.memory.segment-size: 131072
I think the maximum possible value given Netty's "writeBufferHighWaterMark" parameter is 65536.
There might be a way to tune Netty's parameters but I don't know how to do that. Maybe Ufuk (in CC) knows better.
Is there any particular reason why want to increase the network buffer size instead of keeping the default?
Did you find that the default parameter gives insufficient performance?
Best, Fabian
2016-12-26 20:52 GMT+01:00 <ka...@verizon.com>>:
Hi,
I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error frequently ... and those task managers not processing messages after that error.
What could be wrong in my configuration? What I need to do to avoid this error.?
Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072
2016-12-26 14:26:06,548| |WARN io.netty.bootstrap.ServerBootstrap - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812<http://138.83.31.4:60812> => /138.83.31.9:41304<http://138.83.31.9:41304>]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073
at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Thanks
Kanagaraj Vengidasamy
RTCI
7701 E Telecom PKWY
Temple Terrace, FL 33637
O 813.978.4372 | M 813.455.9757
Re: [E] writeBufferLowWaterMark cannot be greater than
writeBufferHighWaterMark error
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Kanagaraj,
I would assume that the issue is caused by this configuration parameter:
taskmanager.memory.segment-size: 131072
I think the maximum possible value given Netty's "writeBufferHighWaterMark"
parameter is 65536.
There might be a way to tune Netty's parameters but I don't know how to do
that. Maybe Ufuk (in CC) knows better.
Is there any particular reason why want to increase the network buffer size
instead of keeping the default?
Did you find that the default parameter gives insufficient performance?
Best, Fabian
2016-12-26 20:52 GMT+01:00 <ka...@verizon.com>:
>
> Hi,
>
> I am getting writeBufferLowWaterMark cannot be greater than
> writeBufferHighWaterMark error frequently ... and those task managers not
> processing messages after that error.
> What could be wrong in my configuration? What I need to do to avoid this
> error.?
>
> Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8
> slots)
> taskmanager.heap.mb: 4096
> taskmanager.numberOfTaskSlots: 8
> taskmanager.network.numberOfBuffers: 22000
> taskmanager.memory.segment-size: 131072
>
>
> 2016-12-26 14:26:06,548| |WARN io.netty.bootstrap.ServerBootstrap -
> Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /
> 138.83.31.9:41304]
> java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be
> greater than writeBufferHighWaterMark (65536): 131073
> at io.netty.channel.DefaultChannelConfig.
> setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
> at io.netty.channel.socket.DefaultSocketChannelConfig.
> setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
> at io.netty.channel.socket.DefaultSocketChannelConfig.
> setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
> at io.netty.channel.DefaultChannelConfig.setOption(
> DefaultChannelConfig.java:183)
> at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(
> DefaultSocketChannelConfig.java:121)
> at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.
> channelRead(ServerBootstrap.java:238)
> at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:324)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(
> DefaultChannelPipeline.java:847)
> at io.netty.channel.nio.AbstractNioMessageChannel$
> NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(
> NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
> NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
> NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks
> Kanagaraj Vengidasamy
> RTCI
>
> 7701 E Telecom PKWY
> Temple Terrace, FL 33637
>
> O 813.978.4372 | M 813.455.9757
>
>
>
>