You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pavel Sapozhnikov <pa...@gmail.com> on 2019/06/08 23:28:47 UTC
Re: Kafka streams in Kubernetes
I suggest take a look at Strimzi project https://strimzi.io/
Kafka operator deployed in Kubernetes environment.
On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <mp...@hpe.com> wrote:
> Hi,
>
> I have read several articles about this topic. We are soon going to deploy
> our streaming apps inside k8s. My understanding from reading these articles
> is that stateful set in k8s is not mandatory as the application can rebuild
> its state if the state store is not present. Can people share their
> experience or recommendation when it comes to deploying the streaming apps
> on k8s ?
>
> Also, let us say the application is using a tumbling window of 5 mts. When
> an application restarts, is it correct to say that it has to re-build the
> state only for that 5 minute window for the partitions that it was handling
> before. I had an instance of such a restart where it was running a long
> time in REBALANCE which makes me think that my understanding is incorrect.
> In this case, the state store was available during the restart. Can someone
> clarify ?
>
> Thanks
> Mohan
>
>
Re: Kafka streams in Kubernetes
Posted by "Parthasarathy, Mohan" <mp...@hpe.com>.
Pavel,
Thanks for the pointer. I will take a look.
-mohan
On 6/8/19, 4:29 PM, "Pavel Sapozhnikov" <pa...@gmail.com> wrote:
I suggest take a look at Strimzi project https://strimzi.io/
Kafka operator deployed in Kubernetes environment.
On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <mp...@hpe.com> wrote:
> Hi,
>
> I have read several articles about this topic. We are soon going to deploy
> our streaming apps inside k8s. My understanding from reading these articles
> is that stateful set in k8s is not mandatory as the application can rebuild
> its state if the state store is not present. Can people share their
> experience or recommendation when it comes to deploying the streaming apps
> on k8s ?
>
> Also, let us say the application is using a tumbling window of 5 mts. When
> an application restarts, is it correct to say that it has to re-build the
> state only for that 5 minute window for the partitions that it was handling
> before. I had an instance of such a restart where it was running a long
> time in REBALANCE which makes me think that my understanding is incorrect.
> In this case, the state store was available during the restart. Can someone
> clarify ?
>
> Thanks
> Mohan
>
>
Re: Kafka streams in Kubernetes
Posted by Scott Reynolds <sr...@twilio.com.INVALID>.
We have been giving this a bunch of thought lately. We attempted to
replace PARTITION_ASSIGNMENT_STRATEGY_CONFIG
with our implementation that hooks into our deployment service. The idea is
simple, the new deployment gets *Standby tasks assigned to them until they
are caught up*. Once they are caught up, our deployment service takes the
older deployment down and the new deployment takes over all the active
tasks. We think it is possible to implement but there are large amount of
cohesion between consumers and the stream setup that we are wading through.
We ended up writing:
1. AssignmentInfo
2. SubscriptionInfo
3. StickyStandbyTaskAssignor
4. PartitionAssigner
All of which were largely copy and paste. Hoping we get to pick it back up
soon and able to find a way to make this cleaner. Everything is coupled
together in a pretty tight ball of goo today.
On Mon, Jun 10, 2019 at 11:55 AM Parthasarathy, Mohan <mp...@hpe.com>
wrote:
> Matt,
>
> I read your email again and this one that you point out:
>
> > What you also need to take into account is, how often topics are
> > compacted, and how large the segment size is, because the active
> segment
> > is not subject to compaction.
>
> Are you saying that compaction affects the rebuilding time ? Sorry, I am
> not sure I understand what you meant by this in the current context.
>
> -mohan
>
>
> On 6/10/19, 10:22 AM, "Parthasarathy, Mohan" <mp...@hpe.com> wrote:
>
> Thanks. That helps me understand why recreating state might take time.
>
> -mohan
>
>
> On 6/9/19, 11:50 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
>
> By default, Kafka Streams does not "close" windows.
>
> To handle out-of-order data, windows are maintained until their
> retention time passed, and are updated each time an out-of-order
> record
> arrives (even if window-end time passed).
>
> Cf
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_38935904_how-2Dto-2Dsend-2Dfinal-2Dkafka-2Dstreams-2Daggregation-2Dresult-2Dof-2Da-2Dtime-2Dwindowed-2Dktable&d=DwIGaQ&c=x_Y1Lz9GyeGp2OvBCa_eow&r=ChXZJWKniTslJvQGptpIW7qAh4kkrpgYSer_wfh4G5w&m=NPSMv1cPOmhxttSznyQJ_gbzxa1AHCJGXyS53vGbCFo&s=YUXTr0Z0Neki2SOQESJ6oLUelyCtlfoHyw-CMvWJKkI&e=
>
> for details; note, that using `suppress()` and setting a
> grace-period
> change the default behavior.
>
> Hence, you usually want to keep windows around until you are sure
> that
> no more out-of-order records for a window may arrive (what is
> usually
> some time after the window end time). If 1 day is too large, you
> can of
> course reduce the retention time accordingly.
>
> If you use Interactive Queries, you would need to set retention
> time
> large enough to allow your application to query the state. For this
> case, retention time might be higher to keep state around for
> serving
> queries, even if you know it won't be updated any longer.
>
> For Kafka Streams, using 1 day as default provides a good
> out-of-the-box
> experience. For production deployments, change the retention time
> base
> on the need of the application make sense of course.
>
> There is also one more config you might want to consider:
> `windowstore.changelog.additional.retention.ms` -- it's 1 day by
> default, too, and you might want to reduce it to reduce the amount
> of
> data that is restored.
>
>
> In general, there in nothing wrong with using stateful sets
> though, and
> for large state, it's recommended to avoid long recovery times.
>
>
>
> -Matthias
>
>
> On 6/9/19 2:59 PM, Parthasarathy, Mohan wrote:
> > Matt,
> >
> > Thanks for your response. I agree with you that there is no easy
> way to answer this. I was trying to see what others experience is which
> could simply be "Don't bother, in practice stateful set is better".
> >
> > Could you explain as to why there has to be more state than the
> window size ? In a running application, as the data is being processed from
> a topic, there is state being created depending on the stateful primitives.
> As the window is closed, this state is not needed and I can see why grace
> period has to be taken into account ? So, when would you need it for the
> whole store retention time ? Could you clarify ?
> >
> > Thanks
> > Mohan
> >
> > On 6/8/19, 11:18 PM, "Matthias J. Sax" <ma...@confluent.io>
> wrote:
> >
> > If depends how much state you need to restore and how much
> restore-time
> > you can accept in your application.
> >
> > The amount of data that needs to be restored, does not
> depend on the
> > window-size, but the store retention time (default 1 day,
> configurable
> > via `Materialized#withRetention()`). The window size (and
> grace period,
> > if case you use one) is a lower bound for the configurable
> retention
> > time though, ie, retention time >= grace-period >=
> window-size.
> >
> > What you also need to take into account is, how often topics
> are
> > compacted, and how large the segment size is, because the
> active segment
> > is not subject to compaction.
> >
> > It's always hard to answer a question like this. I would
> recommend to do
> > some testing and benchmark fail over, by manually killing
> some instances
> > to simulate a crash. This should give the best insight --
> tuning the
> > above parameters, you can see, what works for your
> application.
> >
> >
> >
> > -Matthias
> >
> > On 6/8/19 4:28 PM, Pavel Sapozhnikov wrote:
> > > I suggest take a look at Strimzi project
> https://urldefense.proofpoint.com/v2/url?u=https-3A__strimzi.io_&d=DwIGaQ&c=x_Y1Lz9GyeGp2OvBCa_eow&r=ChXZJWKniTslJvQGptpIW7qAh4kkrpgYSer_wfh4G5w&m=NPSMv1cPOmhxttSznyQJ_gbzxa1AHCJGXyS53vGbCFo&s=qu-Z9wpqbvWtwzM1ED2N1U5Rvj2-N_y6iQgGMFGcAT4&e=
>
> > >
> > > Kafka operator deployed in Kubernetes environment.
> > >
> > > On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <
> mparthas@hpe.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I have read several articles about this topic. We are
> soon going to deploy
> > >> our streaming apps inside k8s. My understanding from
> reading these articles
> > >> is that stateful set in k8s is not mandatory as the
> application can rebuild
> > >> its state if the state store is not present. Can people
> share their
> > >> experience or recommendation when it comes to deploying
> the streaming apps
> > >> on k8s ?
> > >>
> > >> Also, let us say the application is using a tumbling
> window of 5 mts. When
> > >> an application restarts, is it correct to say that it has
> to re-build the
> > >> state only for that 5 minute window for the partitions
> that it was handling
> > >> before. I had an instance of such a restart where it was
> running a long
> > >> time in REBALANCE which makes me think that my
> understanding is incorrect.
> > >> In this case, the state store was available during the
> restart. Can someone
> > >> clarify ?
> > >>
> > >> Thanks
> > >> Mohan
> > >>
> > >>
> > >
> >
> >
> >
>
>
>
>
>
>
Re: Kafka streams in Kubernetes
Posted by "Matthias J. Sax" <ma...@confluent.io>.
What I try to say is, that compaction is not perfect.
Assuming you have 100000 unique keys, and a message size of 1KB, this
implies that your data set if it's perfectly compacted would be roughly
100MB.
The default segment size is 1GB and the active segment is not compacted.
Hence, if the active segment grows over time, it may contain each key up
10x on average.
Therefore, because Kafka Streams need to consumer the full active
segment on restore it would read up to 1GB of data instead of just 100MB
to recover the state.
Hence, you might want to consider to reduce the segment size and segment
roll internal for this case. It's not recommended to make it too small
though as it will also affect broker performance.
If you have large state with many GB of data, the overhead from the
non-compacted active segment is rather low. However, as you consider to
go with in-memory stores, I would assume that your state is not too
large, and for small state you might hit this case.
-Matthias
On 6/10/19 11:54 AM, Parthasarathy, Mohan wrote:
> Matt,
>
> I read your email again and this one that you point out:
>
> > What you also need to take into account is, how often topics are
> > compacted, and how large the segment size is, because the active segment
> > is not subject to compaction.
>
> Are you saying that compaction affects the rebuilding time ? Sorry, I am not sure I understand what you meant by this in the current context.
>
> -mohan
>
>
> On 6/10/19, 10:22 AM, "Parthasarathy, Mohan" <mp...@hpe.com> wrote:
>
> Thanks. That helps me understand why recreating state might take time.
>
> -mohan
>
>
> On 6/9/19, 11:50 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
>
> By default, Kafka Streams does not "close" windows.
>
> To handle out-of-order data, windows are maintained until their
> retention time passed, and are updated each time an out-of-order record
> arrives (even if window-end time passed).
>
> Cf
> https://stackoverflow.com/questions/38935904/how-to-send-final-kafka-streams-aggregation-result-of-a-time-windowed-ktable
> for details; note, that using `suppress()` and setting a grace-period
> change the default behavior.
>
> Hence, you usually want to keep windows around until you are sure that
> no more out-of-order records for a window may arrive (what is usually
> some time after the window end time). If 1 day is too large, you can of
> course reduce the retention time accordingly.
>
> If you use Interactive Queries, you would need to set retention time
> large enough to allow your application to query the state. For this
> case, retention time might be higher to keep state around for serving
> queries, even if you know it won't be updated any longer.
>
> For Kafka Streams, using 1 day as default provides a good out-of-the-box
> experience. For production deployments, change the retention time base
> on the need of the application make sense of course.
>
> There is also one more config you might want to consider:
> `windowstore.changelog.additional.retention.ms` -- it's 1 day by
> default, too, and you might want to reduce it to reduce the amount of
> data that is restored.
>
>
> In general, there in nothing wrong with using stateful sets though, and
> for large state, it's recommended to avoid long recovery times.
>
>
>
> -Matthias
>
>
> On 6/9/19 2:59 PM, Parthasarathy, Mohan wrote:
> > Matt,
> >
> > Thanks for your response. I agree with you that there is no easy way to answer this. I was trying to see what others experience is which could simply be "Don't bother, in practice stateful set is better".
> >
> > Could you explain as to why there has to be more state than the window size ? In a running application, as the data is being processed from a topic, there is state being created depending on the stateful primitives. As the window is closed, this state is not needed and I can see why grace period has to be taken into account ? So, when would you need it for the whole store retention time ? Could you clarify ?
> >
> > Thanks
> > Mohan
> >
> > On 6/8/19, 11:18 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
> >
> > If depends how much state you need to restore and how much restore-time
> > you can accept in your application.
> >
> > The amount of data that needs to be restored, does not depend on the
> > window-size, but the store retention time (default 1 day, configurable
> > via `Materialized#withRetention()`). The window size (and grace period,
> > if case you use one) is a lower bound for the configurable retention
> > time though, ie, retention time >= grace-period >= window-size.
> >
> > What you also need to take into account is, how often topics are
> > compacted, and how large the segment size is, because the active segment
> > is not subject to compaction.
> >
> > It's always hard to answer a question like this. I would recommend to do
> > some testing and benchmark fail over, by manually killing some instances
> > to simulate a crash. This should give the best insight -- tuning the
> > above parameters, you can see, what works for your application.
> >
> >
> >
> > -Matthias
> >
> > On 6/8/19 4:28 PM, Pavel Sapozhnikov wrote:
> > > I suggest take a look at Strimzi project https://strimzi.io/
> > >
> > > Kafka operator deployed in Kubernetes environment.
> > >
> > > On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <mp...@hpe.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I have read several articles about this topic. We are soon going to deploy
> > >> our streaming apps inside k8s. My understanding from reading these articles
> > >> is that stateful set in k8s is not mandatory as the application can rebuild
> > >> its state if the state store is not present. Can people share their
> > >> experience or recommendation when it comes to deploying the streaming apps
> > >> on k8s ?
> > >>
> > >> Also, let us say the application is using a tumbling window of 5 mts. When
> > >> an application restarts, is it correct to say that it has to re-build the
> > >> state only for that 5 minute window for the partitions that it was handling
> > >> before. I had an instance of such a restart where it was running a long
> > >> time in REBALANCE which makes me think that my understanding is incorrect.
> > >> In this case, the state store was available during the restart. Can someone
> > >> clarify ?
> > >>
> > >> Thanks
> > >> Mohan
> > >>
> > >>
> > >
> >
> >
> >
>
>
>
>
>
Re: Kafka streams in Kubernetes
Posted by "Parthasarathy, Mohan" <mp...@hpe.com>.
Matt,
I read your email again and this one that you point out:
> What you also need to take into account is, how often topics are
> compacted, and how large the segment size is, because the active segment
> is not subject to compaction.
Are you saying that compaction affects the rebuilding time ? Sorry, I am not sure I understand what you meant by this in the current context.
-mohan
On 6/10/19, 10:22 AM, "Parthasarathy, Mohan" <mp...@hpe.com> wrote:
Thanks. That helps me understand why recreating state might take time.
-mohan
On 6/9/19, 11:50 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
By default, Kafka Streams does not "close" windows.
To handle out-of-order data, windows are maintained until their
retention time passed, and are updated each time an out-of-order record
arrives (even if window-end time passed).
Cf
https://stackoverflow.com/questions/38935904/how-to-send-final-kafka-streams-aggregation-result-of-a-time-windowed-ktable
for details; note, that using `suppress()` and setting a grace-period
change the default behavior.
Hence, you usually want to keep windows around until you are sure that
no more out-of-order records for a window may arrive (what is usually
some time after the window end time). If 1 day is too large, you can of
course reduce the retention time accordingly.
If you use Interactive Queries, you would need to set retention time
large enough to allow your application to query the state. For this
case, retention time might be higher to keep state around for serving
queries, even if you know it won't be updated any longer.
For Kafka Streams, using 1 day as default provides a good out-of-the-box
experience. For production deployments, change the retention time base
on the need of the application make sense of course.
There is also one more config you might want to consider:
`windowstore.changelog.additional.retention.ms` -- it's 1 day by
default, too, and you might want to reduce it to reduce the amount of
data that is restored.
In general, there in nothing wrong with using stateful sets though, and
for large state, it's recommended to avoid long recovery times.
-Matthias
On 6/9/19 2:59 PM, Parthasarathy, Mohan wrote:
> Matt,
>
> Thanks for your response. I agree with you that there is no easy way to answer this. I was trying to see what others experience is which could simply be "Don't bother, in practice stateful set is better".
>
> Could you explain as to why there has to be more state than the window size ? In a running application, as the data is being processed from a topic, there is state being created depending on the stateful primitives. As the window is closed, this state is not needed and I can see why grace period has to be taken into account ? So, when would you need it for the whole store retention time ? Could you clarify ?
>
> Thanks
> Mohan
>
> On 6/8/19, 11:18 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
>
> If depends how much state you need to restore and how much restore-time
> you can accept in your application.
>
> The amount of data that needs to be restored, does not depend on the
> window-size, but the store retention time (default 1 day, configurable
> via `Materialized#withRetention()`). The window size (and grace period,
> if case you use one) is a lower bound for the configurable retention
> time though, ie, retention time >= grace-period >= window-size.
>
> What you also need to take into account is, how often topics are
> compacted, and how large the segment size is, because the active segment
> is not subject to compaction.
>
> It's always hard to answer a question like this. I would recommend to do
> some testing and benchmark fail over, by manually killing some instances
> to simulate a crash. This should give the best insight -- tuning the
> above parameters, you can see, what works for your application.
>
>
>
> -Matthias
>
> On 6/8/19 4:28 PM, Pavel Sapozhnikov wrote:
> > I suggest take a look at Strimzi project https://strimzi.io/
> >
> > Kafka operator deployed in Kubernetes environment.
> >
> > On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <mp...@hpe.com> wrote:
> >
> >> Hi,
> >>
> >> I have read several articles about this topic. We are soon going to deploy
> >> our streaming apps inside k8s. My understanding from reading these articles
> >> is that stateful set in k8s is not mandatory as the application can rebuild
> >> its state if the state store is not present. Can people share their
> >> experience or recommendation when it comes to deploying the streaming apps
> >> on k8s ?
> >>
> >> Also, let us say the application is using a tumbling window of 5 mts. When
> >> an application restarts, is it correct to say that it has to re-build the
> >> state only for that 5 minute window for the partitions that it was handling
> >> before. I had an instance of such a restart where it was running a long
> >> time in REBALANCE which makes me think that my understanding is incorrect.
> >> In this case, the state store was available during the restart. Can someone
> >> clarify ?
> >>
> >> Thanks
> >> Mohan
> >>
> >>
> >
>
>
>
Re: Kafka streams in Kubernetes
Posted by "Parthasarathy, Mohan" <mp...@hpe.com>.
Thanks. That helps me understand why recreating state might take time.
-mohan
On 6/9/19, 11:50 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
By default, Kafka Streams does not "close" windows.
To handle out-of-order data, windows are maintained until their
retention time passed, and are updated each time an out-of-order record
arrives (even if window-end time passed).
Cf
https://stackoverflow.com/questions/38935904/how-to-send-final-kafka-streams-aggregation-result-of-a-time-windowed-ktable
for details; note, that using `suppress()` and setting a grace-period
change the default behavior.
Hence, you usually want to keep windows around until you are sure that
no more out-of-order records for a window may arrive (what is usually
some time after the window end time). If 1 day is too large, you can of
course reduce the retention time accordingly.
If you use Interactive Queries, you would need to set retention time
large enough to allow your application to query the state. For this
case, retention time might be higher to keep state around for serving
queries, even if you know it won't be updated any longer.
For Kafka Streams, using 1 day as default provides a good out-of-the-box
experience. For production deployments, change the retention time base
on the need of the application make sense of course.
There is also one more config you might want to consider:
`windowstore.changelog.additional.retention.ms` -- it's 1 day by
default, too, and you might want to reduce it to reduce the amount of
data that is restored.
In general, there in nothing wrong with using stateful sets though, and
for large state, it's recommended to avoid long recovery times.
-Matthias
On 6/9/19 2:59 PM, Parthasarathy, Mohan wrote:
> Matt,
>
> Thanks for your response. I agree with you that there is no easy way to answer this. I was trying to see what others experience is which could simply be "Don't bother, in practice stateful set is better".
>
> Could you explain as to why there has to be more state than the window size ? In a running application, as the data is being processed from a topic, there is state being created depending on the stateful primitives. As the window is closed, this state is not needed and I can see why grace period has to be taken into account ? So, when would you need it for the whole store retention time ? Could you clarify ?
>
> Thanks
> Mohan
>
> On 6/8/19, 11:18 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
>
> If depends how much state you need to restore and how much restore-time
> you can accept in your application.
>
> The amount of data that needs to be restored, does not depend on the
> window-size, but the store retention time (default 1 day, configurable
> via `Materialized#withRetention()`). The window size (and grace period,
> if case you use one) is a lower bound for the configurable retention
> time though, ie, retention time >= grace-period >= window-size.
>
> What you also need to take into account is, how often topics are
> compacted, and how large the segment size is, because the active segment
> is not subject to compaction.
>
> It's always hard to answer a question like this. I would recommend to do
> some testing and benchmark fail over, by manually killing some instances
> to simulate a crash. This should give the best insight -- tuning the
> above parameters, you can see, what works for your application.
>
>
>
> -Matthias
>
> On 6/8/19 4:28 PM, Pavel Sapozhnikov wrote:
> > I suggest take a look at Strimzi project https://strimzi.io/
> >
> > Kafka operator deployed in Kubernetes environment.
> >
> > On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <mp...@hpe.com> wrote:
> >
> >> Hi,
> >>
> >> I have read several articles about this topic. We are soon going to deploy
> >> our streaming apps inside k8s. My understanding from reading these articles
> >> is that stateful set in k8s is not mandatory as the application can rebuild
> >> its state if the state store is not present. Can people share their
> >> experience or recommendation when it comes to deploying the streaming apps
> >> on k8s ?
> >>
> >> Also, let us say the application is using a tumbling window of 5 mts. When
> >> an application restarts, is it correct to say that it has to re-build the
> >> state only for that 5 minute window for the partitions that it was handling
> >> before. I had an instance of such a restart where it was running a long
> >> time in REBALANCE which makes me think that my understanding is incorrect.
> >> In this case, the state store was available during the restart. Can someone
> >> clarify ?
> >>
> >> Thanks
> >> Mohan
> >>
> >>
> >
>
>
>
Re: Kafka streams in Kubernetes
Posted by "Matthias J. Sax" <ma...@confluent.io>.
By default, Kafka Streams does not "close" windows.
To handle out-of-order data, windows are maintained until their
retention time passed, and are updated each time an out-of-order record
arrives (even if window-end time passed).
Cf
https://stackoverflow.com/questions/38935904/how-to-send-final-kafka-streams-aggregation-result-of-a-time-windowed-ktable
for details; note, that using `suppress()` and setting a grace-period
change the default behavior.
Hence, you usually want to keep windows around until you are sure that
no more out-of-order records for a window may arrive (what is usually
some time after the window end time). If 1 day is too large, you can of
course reduce the retention time accordingly.
If you use Interactive Queries, you would need to set retention time
large enough to allow your application to query the state. For this
case, retention time might be higher to keep state around for serving
queries, even if you know it won't be updated any longer.
For Kafka Streams, using 1 day as default provides a good out-of-the-box
experience. For production deployments, change the retention time base
on the need of the application make sense of course.
There is also one more config you might want to consider:
`windowstore.changelog.additional.retention.ms` -- it's 1 day by
default, too, and you might want to reduce it to reduce the amount of
data that is restored.
In general, there in nothing wrong with using stateful sets though, and
for large state, it's recommended to avoid long recovery times.
-Matthias
On 6/9/19 2:59 PM, Parthasarathy, Mohan wrote:
> Matt,
>
> Thanks for your response. I agree with you that there is no easy way to answer this. I was trying to see what others experience is which could simply be "Don't bother, in practice stateful set is better".
>
> Could you explain as to why there has to be more state than the window size ? In a running application, as the data is being processed from a topic, there is state being created depending on the stateful primitives. As the window is closed, this state is not needed and I can see why grace period has to be taken into account ? So, when would you need it for the whole store retention time ? Could you clarify ?
>
> Thanks
> Mohan
>
> On 6/8/19, 11:18 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
>
> If depends how much state you need to restore and how much restore-time
> you can accept in your application.
>
> The amount of data that needs to be restored, does not depend on the
> window-size, but the store retention time (default 1 day, configurable
> via `Materialized#withRetention()`). The window size (and grace period,
> if case you use one) is a lower bound for the configurable retention
> time though, ie, retention time >= grace-period >= window-size.
>
> What you also need to take into account is, how often topics are
> compacted, and how large the segment size is, because the active segment
> is not subject to compaction.
>
> It's always hard to answer a question like this. I would recommend to do
> some testing and benchmark fail over, by manually killing some instances
> to simulate a crash. This should give the best insight -- tuning the
> above parameters, you can see, what works for your application.
>
>
>
> -Matthias
>
> On 6/8/19 4:28 PM, Pavel Sapozhnikov wrote:
> > I suggest take a look at Strimzi project https://strimzi.io/
> >
> > Kafka operator deployed in Kubernetes environment.
> >
> > On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <mp...@hpe.com> wrote:
> >
> >> Hi,
> >>
> >> I have read several articles about this topic. We are soon going to deploy
> >> our streaming apps inside k8s. My understanding from reading these articles
> >> is that stateful set in k8s is not mandatory as the application can rebuild
> >> its state if the state store is not present. Can people share their
> >> experience or recommendation when it comes to deploying the streaming apps
> >> on k8s ?
> >>
> >> Also, let us say the application is using a tumbling window of 5 mts. When
> >> an application restarts, is it correct to say that it has to re-build the
> >> state only for that 5 minute window for the partitions that it was handling
> >> before. I had an instance of such a restart where it was running a long
> >> time in REBALANCE which makes me think that my understanding is incorrect.
> >> In this case, the state store was available during the restart. Can someone
> >> clarify ?
> >>
> >> Thanks
> >> Mohan
> >>
> >>
> >
>
>
>
Re: Kafka streams in Kubernetes
Posted by "Parthasarathy, Mohan" <mp...@hpe.com>.
Matt,
Thanks for your response. I agree with you that there is no easy way to answer this. I was trying to see what others experience is which could simply be "Don't bother, in practice stateful set is better".
Could you explain as to why there has to be more state than the window size ? In a running application, as the data is being processed from a topic, there is state being created depending on the stateful primitives. As the window is closed, this state is not needed and I can see why grace period has to be taken into account ? So, when would you need it for the whole store retention time ? Could you clarify ?
Thanks
Mohan
On 6/8/19, 11:18 PM, "Matthias J. Sax" <ma...@confluent.io> wrote:
If depends how much state you need to restore and how much restore-time
you can accept in your application.
The amount of data that needs to be restored, does not depend on the
window-size, but the store retention time (default 1 day, configurable
via `Materialized#withRetention()`). The window size (and grace period,
if case you use one) is a lower bound for the configurable retention
time though, ie, retention time >= grace-period >= window-size.
What you also need to take into account is, how often topics are
compacted, and how large the segment size is, because the active segment
is not subject to compaction.
It's always hard to answer a question like this. I would recommend to do
some testing and benchmark fail over, by manually killing some instances
to simulate a crash. This should give the best insight -- tuning the
above parameters, you can see, what works for your application.
-Matthias
On 6/8/19 4:28 PM, Pavel Sapozhnikov wrote:
> I suggest take a look at Strimzi project https://strimzi.io/
>
> Kafka operator deployed in Kubernetes environment.
>
> On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <mp...@hpe.com> wrote:
>
>> Hi,
>>
>> I have read several articles about this topic. We are soon going to deploy
>> our streaming apps inside k8s. My understanding from reading these articles
>> is that stateful set in k8s is not mandatory as the application can rebuild
>> its state if the state store is not present. Can people share their
>> experience or recommendation when it comes to deploying the streaming apps
>> on k8s ?
>>
>> Also, let us say the application is using a tumbling window of 5 mts. When
>> an application restarts, is it correct to say that it has to re-build the
>> state only for that 5 minute window for the partitions that it was handling
>> before. I had an instance of such a restart where it was running a long
>> time in REBALANCE which makes me think that my understanding is incorrect.
>> In this case, the state store was available during the restart. Can someone
>> clarify ?
>>
>> Thanks
>> Mohan
>>
>>
>
Re: Kafka streams in Kubernetes
Posted by "Matthias J. Sax" <ma...@confluent.io>.
If depends how much state you need to restore and how much restore-time
you can accept in your application.
The amount of data that needs to be restored, does not depend on the
window-size, but the store retention time (default 1 day, configurable
via `Materialized#withRetention()`). The window size (and grace period,
if case you use one) is a lower bound for the configurable retention
time though, ie, retention time >= grace-period >= window-size.
What you also need to take into account is, how often topics are
compacted, and how large the segment size is, because the active segment
is not subject to compaction.
It's always hard to answer a question like this. I would recommend to do
some testing and benchmark fail over, by manually killing some instances
to simulate a crash. This should give the best insight -- tuning the
above parameters, you can see, what works for your application.
-Matthias
On 6/8/19 4:28 PM, Pavel Sapozhnikov wrote:
> I suggest take a look at Strimzi project https://strimzi.io/
>
> Kafka operator deployed in Kubernetes environment.
>
> On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan <mp...@hpe.com> wrote:
>
>> Hi,
>>
>> I have read several articles about this topic. We are soon going to deploy
>> our streaming apps inside k8s. My understanding from reading these articles
>> is that stateful set in k8s is not mandatory as the application can rebuild
>> its state if the state store is not present. Can people share their
>> experience or recommendation when it comes to deploying the streaming apps
>> on k8s ?
>>
>> Also, let us say the application is using a tumbling window of 5 mts. When
>> an application restarts, is it correct to say that it has to re-build the
>> state only for that 5 minute window for the partitions that it was handling
>> before. I had an instance of such a restart where it was running a long
>> time in REBALANCE which makes me think that my understanding is incorrect.
>> In this case, the state store was available during the restart. Can someone
>> clarify ?
>>
>> Thanks
>> Mohan
>>
>>
>