You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Hamza HACHANI <ha...@supcom.tn> on 2016/11/04 08:42:37 UTC
Control results coming from windows
Hi,
I implemented window using Stream DSL.But results does get out from the window anytime a new data come. Is there anyway to make result get out from the window just at the end of the size of the window not like this way.
In other words, I do need to control the flow of data getting out of the window.I'm only interested in the final result not the intermediate ones.
Thanks in advance,
Hamza
RE: Control results coming from windows
Posted by Hamza HACHANI <ha...@supcom.tn>.
OK Thanks Damian.
Have a nice day.
Hamza
________________________________
De : Damian Guy <da...@gmail.com>
Envoyé : vendredi 4 novembre 2016 00:58:36
À : users@kafka.apache.org
Objet : Re: Control results coming from windows
Hi Hamza,
I'm not sure what you mean in the first sentence? There are some breaking
API changes form 0.10.0 -> 0.10.1, so you may need to change some code. I'd
also suggest you thoroughly test with the new version to ensure there are
no regressions.
There is a known issue with caching that may or may not effect you. It
depends on your topology. https://issues.apache.org/jira/browse/KAFKA-4311 -
there is a fix for it that will hopefully be merged to trunk and 0.10.1
soon, however i don't have any timeline for when/if a 0.10.1.1 release will
be done.
The config parameter is StreamsConfig.CACHE_MAX_BYTES_BUFFERING. You should
also take a look at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams
Thanks,
Damian
On Fri, 4 Nov 2016 at 09:44 Hamza HACHANI <ha...@supcom.tn> wrote:
> Hi Damian,
>
>
> If i would pass tp the 0.10.1. Is this possible without impacting the
> applications that i've already done ?
>
> If yes then which parameters are needed for deduplication with caching.
> Like this you would save me time looking for it.
>
>
> Thanks.
>
> Hamza
>
> ________________________________
> De : Damian Guy <da...@gmail.com>
> Envoyé : jeudi 3 novembre 2016 21:32:52
> À : users@kafka.apache.org
> Objet : Re: Control results coming from windows
>
> Hi Hamza,
>
> If you are using version 0.10.0, then there is no way of controlling this.
> In 0.10.1 we do some deduplication with caching, but you should still
> expect multiple results for a window.
>
> Thanks,
> Damian
>
> On Fri, 4 Nov 2016 at 08:42 Hamza HACHANI <ha...@supcom.tn> wrote:
>
> > Hi,
> >
> >
> > I implemented window using Stream DSL.But results does get out from the
> > window anytime a new data come. Is there anyway to make result get out
> > from the window just at the end of the size of the window not like this
> way.
> >
> > In other words, I do need to control the flow of data getting out of the
> > window.I'm only interested in the final result not the intermediate ones.
> >
> >
> > Thanks in advance,
> >
> > Hamza
> >
>
Re: Control results coming from windows
Posted by Damian Guy <da...@gmail.com>.
Hi Hamza,
I'm not sure what you mean in the first sentence? There are some breaking
API changes form 0.10.0 -> 0.10.1, so you may need to change some code. I'd
also suggest you thoroughly test with the new version to ensure there are
no regressions.
There is a known issue with caching that may or may not effect you. It
depends on your topology. https://issues.apache.org/jira/browse/KAFKA-4311 -
there is a fix for it that will hopefully be merged to trunk and 0.10.1
soon, however i don't have any timeline for when/if a 0.10.1.1 release will
be done.
The config parameter is StreamsConfig.CACHE_MAX_BYTES_BUFFERING. You should
also take a look at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams
Thanks,
Damian
On Fri, 4 Nov 2016 at 09:44 Hamza HACHANI <ha...@supcom.tn> wrote:
> Hi Damian,
>
>
> If i would pass tp the 0.10.1. Is this possible without impacting the
> applications that i've already done ?
>
> If yes then which parameters are needed for deduplication with caching.
> Like this you would save me time looking for it.
>
>
> Thanks.
>
> Hamza
>
> ________________________________
> De : Damian Guy <da...@gmail.com>
> Envoyé : jeudi 3 novembre 2016 21:32:52
> À : users@kafka.apache.org
> Objet : Re: Control results coming from windows
>
> Hi Hamza,
>
> If you are using version 0.10.0, then there is no way of controlling this.
> In 0.10.1 we do some deduplication with caching, but you should still
> expect multiple results for a window.
>
> Thanks,
> Damian
>
> On Fri, 4 Nov 2016 at 08:42 Hamza HACHANI <ha...@supcom.tn> wrote:
>
> > Hi,
> >
> >
> > I implemented window using Stream DSL.But results does get out from the
> > window anytime a new data come. Is there anyway to make result get out
> > from the window just at the end of the size of the window not like this
> way.
> >
> > In other words, I do need to control the flow of data getting out of the
> > window.I'm only interested in the final result not the intermediate ones.
> >
> >
> > Thanks in advance,
> >
> > Hamza
> >
>
RE: Control results coming from windows
Posted by Hamza HACHANI <ha...@supcom.tn>.
Hi Damian,
If i would pass tp the 0.10.1. Is this possible without impacting the applications that i've already done ?
If yes then which parameters are needed for deduplication with caching. Like this you would save me time looking for it.
Thanks.
Hamza
________________________________
De : Damian Guy <da...@gmail.com>
Envoyé : jeudi 3 novembre 2016 21:32:52
À : users@kafka.apache.org
Objet : Re: Control results coming from windows
Hi Hamza,
If you are using version 0.10.0, then there is no way of controlling this.
In 0.10.1 we do some deduplication with caching, but you should still
expect multiple results for a window.
Thanks,
Damian
On Fri, 4 Nov 2016 at 08:42 Hamza HACHANI <ha...@supcom.tn> wrote:
> Hi,
>
>
> I implemented window using Stream DSL.But results does get out from the
> window anytime a new data come. Is there anyway to make result get out
> from the window just at the end of the size of the window not like this way.
>
> In other words, I do need to control the flow of data getting out of the
> window.I'm only interested in the final result not the intermediate ones.
>
>
> Thanks in advance,
>
> Hamza
>
Re: Control results coming from windows
Posted by Damian Guy <da...@gmail.com>.
Hi Hamza,
If you are using version 0.10.0, then there is no way of controlling this.
In 0.10.1 we do some deduplication with caching, but you should still
expect multiple results for a window.
Thanks,
Damian
On Fri, 4 Nov 2016 at 08:42 Hamza HACHANI <ha...@supcom.tn> wrote:
> Hi,
>
>
> I implemented window using Stream DSL.But results does get out from the
> window anytime a new data come. Is there anyway to make result get out
> from the window just at the end of the size of the window not like this way.
>
> In other words, I do need to control the flow of data getting out of the
> window.I'm only interested in the final result not the intermediate ones.
>
>
> Thanks in advance,
>
> Hamza
>