You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Andrew Ge Wu <an...@eniro.com> on 2016/04/07 15:04:43 UTC
Limit buffer size for a job
Hi guys
We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer.
Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records?
Any suggestion? thanks!
Andrew
--
Confidentiality Notice: This e-mail transmission may contain confidential
or legally privileged information that is intended only for the individual
or entity named in the e-mail address. If you are not the intended
recipient, you are hereby notified that any disclosure, copying,
distribution, or reliance upon the contents of this e-mail is strictly
prohibited and may be unlawful. If you have received this e-mail in error,
please notify the sender immediately by return e-mail and delete all copies
of this message.
Re: Limit buffer size for a job
Posted by Stephan Ewen <se...@apache.org>.
You can reduce Flink's internal network buffering by adjusting the total
number of network buffers.
See
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#configuring-the-network-buffers
That should also make a difference.
On Wed, Apr 13, 2016 at 3:57 PM, Andrew Ge Wu <an...@eniro.com>
wrote:
> Thanks guys for the explanation, I will give it a try.
> After all buffers are filled, the back pressure did it’s job, it works so
> far so good, but I will defiantly give a try to control the latency.
>
>
> Thanks again!
>
>
> Andrew
>
>
>
>
> On 11 Apr 2016, at 18:19, Stephan Ewen <se...@apache.org> wrote:
>
> Hi!
>
> Ufuk's suggestion explains how to buffer less between Flink operators.
>
> Is that what you were looking for, or are you looking for a way to fetch
> more fine grained in the source from the message queue?
> What type of source are you using?
>
> Greetings,
> Stephan
>
>
>
>
> On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
>> Hey Andrew,
>>
>> take a look at this here:
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency
>>
>> Does this help?
>>
>> – Ufuk
>>
>> On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <an...@eniro.com>
>> wrote:
>> > Hi guys
>> >
>> > We have a prioritized queue, where high priority item can jump the
>> queue and we do not want to cache too much record in the buffer.
>> > Is there a way to configure my streaming source to use less buffer? so
>> source always fetch and get latest high prio records?
>> >
>> > Any suggestion? thanks!
>> >
>> >
>> > Andrew
>> > --
>> > Confidentiality Notice: This e-mail transmission may contain
>> confidential
>> > or legally privileged information that is intended only for the
>> individual
>> > or entity named in the e-mail address. If you are not the intended
>> > recipient, you are hereby notified that any disclosure, copying,
>> > distribution, or reliance upon the contents of this e-mail is strictly
>> > prohibited and may be unlawful. If you have received this e-mail in
>> error,
>> > please notify the sender immediately by return e-mail and delete all
>> copies
>> > of this message.
>>
>
>
>
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.
>
Re: Limit buffer size for a job
Posted by Andrew Ge Wu <an...@eniro.com>.
Thanks guys for the explanation, I will give it a try.
After all buffers are filled, the back pressure did it’s job, it works so far so good, but I will defiantly give a try to control the latency.
Thanks again!
Andrew
> On 11 Apr 2016, at 18:19, Stephan Ewen <se...@apache.org> wrote:
>
> Hi!
>
> Ufuk's suggestion explains how to buffer less between Flink operators.
>
> Is that what you were looking for, or are you looking for a way to fetch more fine grained in the source from the message queue?
> What type of source are you using?
>
> Greetings,
> Stephan
>
>
>
>
> On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi <uce@apache.org <ma...@apache.org>> wrote:
> Hey Andrew,
>
> take a look at this here:
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency <https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency>
>
> Does this help?
>
> – Ufuk
>
> On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <andrew.ge-wu@eniro.com <ma...@eniro.com>> wrote:
> > Hi guys
> >
> > We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer.
> > Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records?
> >
> > Any suggestion? thanks!
> >
> >
> > Andrew
> > --
> > Confidentiality Notice: This e-mail transmission may contain confidential
> > or legally privileged information that is intended only for the individual
> > or entity named in the e-mail address. If you are not the intended
> > recipient, you are hereby notified that any disclosure, copying,
> > distribution, or reliance upon the contents of this e-mail is strictly
> > prohibited and may be unlawful. If you have received this e-mail in error,
> > please notify the sender immediately by return e-mail and delete all copies
> > of this message.
>
--
Confidentiality Notice: This e-mail transmission may contain confidential
or legally privileged information that is intended only for the individual
or entity named in the e-mail address. If you are not the intended
recipient, you are hereby notified that any disclosure, copying,
distribution, or reliance upon the contents of this e-mail is strictly
prohibited and may be unlawful. If you have received this e-mail in error,
please notify the sender immediately by return e-mail and delete all copies
of this message.
Re: Limit buffer size for a job
Posted by Stephan Ewen <se...@apache.org>.
Hi!
Ufuk's suggestion explains how to buffer less between Flink operators.
Is that what you were looking for, or are you looking for a way to fetch
more fine grained in the source from the message queue?
What type of source are you using?
Greetings,
Stephan
On Mon, Apr 11, 2016 at 5:02 PM, Ufuk Celebi <uc...@apache.org> wrote:
> Hey Andrew,
>
> take a look at this here:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency
>
> Does this help?
>
> – Ufuk
>
> On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <an...@eniro.com>
> wrote:
> > Hi guys
> >
> > We have a prioritized queue, where high priority item can jump the queue
> and we do not want to cache too much record in the buffer.
> > Is there a way to configure my streaming source to use less buffer? so
> source always fetch and get latest high prio records?
> >
> > Any suggestion? thanks!
> >
> >
> > Andrew
> > --
> > Confidentiality Notice: This e-mail transmission may contain confidential
> > or legally privileged information that is intended only for the
> individual
> > or entity named in the e-mail address. If you are not the intended
> > recipient, you are hereby notified that any disclosure, copying,
> > distribution, or reliance upon the contents of this e-mail is strictly
> > prohibited and may be unlawful. If you have received this e-mail in
> error,
> > please notify the sender immediately by return e-mail and delete all
> copies
> > of this message.
>
Re: Limit buffer size for a job
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Andrew,
take a look at this here:
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html#controlling-latency
Does this help?
– Ufuk
On Thu, Apr 7, 2016 at 3:04 PM, Andrew Ge Wu <an...@eniro.com> wrote:
> Hi guys
>
> We have a prioritized queue, where high priority item can jump the queue and we do not want to cache too much record in the buffer.
> Is there a way to configure my streaming source to use less buffer? so source always fetch and get latest high prio records?
>
> Any suggestion? thanks!
>
>
> Andrew
> --
> Confidentiality Notice: This e-mail transmission may contain confidential
> or legally privileged information that is intended only for the individual
> or entity named in the e-mail address. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or reliance upon the contents of this e-mail is strictly
> prohibited and may be unlawful. If you have received this e-mail in error,
> please notify the sender immediately by return e-mail and delete all copies
> of this message.