You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ian Duffy <ia...@ianduffy.ie> on 2017/03/10 09:05:55 UTC
~20gb of kafka-streams state, unexpected?
Hi All,
I'm doing a groupBy and reduce on a kstream which results in a state store
being created.
This state store is growing to be massive, its filled up a 20gb drive. This
feels very unexpected. Is there some cleanup or flushing process for the
state stores that I'm missing or is such a large size expected?
The topic in question has 96 partitions and the state is about ~200mb
average for each one.
175M 1_0
266M 1_1
164M 1_10
177M 1_11
142M 1_12
271M 1_13
158M 1_14
280M 1_15
286M 1_16
181M 1_17
185M 1_18
187M 1_19
281M 1_2
278M 1_20
188M 1_21
262M 1_22
166M 1_23
177M 1_24
268M 1_25
264M 1_26
147M 1_27
179M 1_28
276M 1_29
177M 1_3
157M 1_30
137M 1_31
247M 1_32
275M 1_33
169M 1_34
267M 1_35
283M 1_36
171M 1_37
166M 1_38
277M 1_39
160M 1_4
273M 1_40
278M 1_41
279M 1_42
170M 1_43
139M 1_44
272M 1_45
179M 1_46
283M 1_47
263M 1_48
267M 1_49
181M 1_5
282M 1_50
166M 1_51
161M 1_52
176M 1_53
152M 1_54
172M 1_55
148M 1_56
268M 1_57
144M 1_58
177M 1_59
271M 1_6
279M 1_60
266M 1_61
194M 1_62
177M 1_63
267M 1_64
177M 1_65
271M 1_66
175M 1_67
168M 1_68
140M 1_69
175M 1_7
173M 1_70
179M 1_71
178M 1_72
166M 1_73
180M 1_74
177M 1_75
276M 1_76
177M 1_77
162M 1_78
266M 1_79
194M 1_8
158M 1_80
187M 1_81
162M 1_82
163M 1_83
177M 1_84
286M 1_85
165M 1_86
171M 1_87
162M 1_88
179M 1_89
145M 1_9
166M 1_90
190M 1_91
159M 1_92
284M 1_93
172M 1_94
149M 1_95
Re: ~20gb of kafka-streams state, unexpected?
Posted by Michael Noll <mi...@confluent.io>.
In addition to what Eno already mentioned here's some quick feedback:
- Only for reference, I'd add that 20GB of state is not necessarily
"massive" in absolute terms. I have talked to users whose apps manage much
more state than that (1-2 orders of magnitude more). Whether or not 20 GB
is massive for your use case is a different question of course,
particularly if you expected much less than 20GB. ;-)
- One further option would be to use probabilistic data structures instead
of the default RocksDB or in-memory key-value stores. I put up a POC that
demonstrates how to peform probabilistic counting with a Count-Min Sketch
backed state store [1]. I think a similar approach would also work for
e.g. Bloomfilters, which might be one potential solution to downsize your
de-duplication problem.
Best,
Michael
[1] https://github.com/confluentinc/examples/pull/100
On Fri, Mar 10, 2017 at 12:22 PM, Eno Thereska <en...@gmail.com>
wrote:
> It’s not necessarily the wrong tool since deduplication is a standard
> scenario, but just setting expectations. If you have enough memory I wonder
> if it would make sense to do it all in-memory with an in-memory store.
> Depends on whether disk or memory space is at a premium.
>
> Thanks
> Eno
>
> > On Mar 10, 2017, at 11:05 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
> >
> > Hi Eno,
> >
> > Thanks for the fast response.
> >
> > We are doing a deduplication process here, so yes you are correct the
> keys
> > are normally unique. Sounds like a wrong tool for the job issue on my
> end.
> >
> > Thanks for your input here.
> >
> >
> >
> > On 10 March 2017 at 10:59, Eno Thereska <en...@gmail.com> wrote:
> >
> >> Hi Ian,
> >>
> >> Sounds like you have a total topic size of ~20GB (96 partitions x
> 200mb).
> >> If most keys are unique then group and reduce might not be as effective
> in
> >> grouping/reducing. Can you comment on the key distribution? Are most
> keys
> >> unique? Or do you expect lots of keys to be the same in the topic?
> >>
> >> Thanks
> >> Eno
> >>
> >>
> >>> On Mar 10, 2017, at 9:05 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I'm doing a groupBy and reduce on a kstream which results in a state
> >> store
> >>> being created.
> >>>
> >>> This state store is growing to be massive, its filled up a 20gb drive.
> >> This
> >>> feels very unexpected. Is there some cleanup or flushing process for
> the
> >>> state stores that I'm missing or is such a large size expected?
> >>>
> >>> The topic in question has 96 partitions and the state is about ~200mb
> >>> average for each one.
> >>>
> >>> 175M 1_0
> >>> 266M 1_1
> >>> 164M 1_10
> >>> 177M 1_11
> >>> 142M 1_12
> >>> 271M 1_13
> >>> 158M 1_14
> >>> 280M 1_15
> >>> 286M 1_16
> >>> 181M 1_17
> >>> 185M 1_18
> >>> 187M 1_19
> >>> 281M 1_2
> >>> 278M 1_20
> >>> 188M 1_21
> >>> 262M 1_22
> >>> 166M 1_23
> >>> 177M 1_24
> >>> 268M 1_25
> >>> 264M 1_26
> >>> 147M 1_27
> >>> 179M 1_28
> >>> 276M 1_29
> >>> 177M 1_3
> >>> 157M 1_30
> >>> 137M 1_31
> >>> 247M 1_32
> >>> 275M 1_33
> >>> 169M 1_34
> >>> 267M 1_35
> >>> 283M 1_36
> >>> 171M 1_37
> >>> 166M 1_38
> >>> 277M 1_39
> >>> 160M 1_4
> >>> 273M 1_40
> >>> 278M 1_41
> >>> 279M 1_42
> >>> 170M 1_43
> >>> 139M 1_44
> >>> 272M 1_45
> >>> 179M 1_46
> >>> 283M 1_47
> >>> 263M 1_48
> >>> 267M 1_49
> >>> 181M 1_5
> >>> 282M 1_50
> >>> 166M 1_51
> >>> 161M 1_52
> >>> 176M 1_53
> >>> 152M 1_54
> >>> 172M 1_55
> >>> 148M 1_56
> >>> 268M 1_57
> >>> 144M 1_58
> >>> 177M 1_59
> >>> 271M 1_6
> >>> 279M 1_60
> >>> 266M 1_61
> >>> 194M 1_62
> >>> 177M 1_63
> >>> 267M 1_64
> >>> 177M 1_65
> >>> 271M 1_66
> >>> 175M 1_67
> >>> 168M 1_68
> >>> 140M 1_69
> >>> 175M 1_7
> >>> 173M 1_70
> >>> 179M 1_71
> >>> 178M 1_72
> >>> 166M 1_73
> >>> 180M 1_74
> >>> 177M 1_75
> >>> 276M 1_76
> >>> 177M 1_77
> >>> 162M 1_78
> >>> 266M 1_79
> >>> 194M 1_8
> >>> 158M 1_80
> >>> 187M 1_81
> >>> 162M 1_82
> >>> 163M 1_83
> >>> 177M 1_84
> >>> 286M 1_85
> >>> 165M 1_86
> >>> 171M 1_87
> >>> 162M 1_88
> >>> 179M 1_89
> >>> 145M 1_9
> >>> 166M 1_90
> >>> 190M 1_91
> >>> 159M 1_92
> >>> 284M 1_93
> >>> 172M 1_94
> >>> 149M 1_95
> >>
> >>
>
>
Re: ~20gb of kafka-streams state, unexpected?
Posted by Eno Thereska <en...@gmail.com>.
It’s not necessarily the wrong tool since deduplication is a standard scenario, but just setting expectations. If you have enough memory I wonder if it would make sense to do it all in-memory with an in-memory store. Depends on whether disk or memory space is at a premium.
Thanks
Eno
> On Mar 10, 2017, at 11:05 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
>
> Hi Eno,
>
> Thanks for the fast response.
>
> We are doing a deduplication process here, so yes you are correct the keys
> are normally unique. Sounds like a wrong tool for the job issue on my end.
>
> Thanks for your input here.
>
>
>
> On 10 March 2017 at 10:59, Eno Thereska <en...@gmail.com> wrote:
>
>> Hi Ian,
>>
>> Sounds like you have a total topic size of ~20GB (96 partitions x 200mb).
>> If most keys are unique then group and reduce might not be as effective in
>> grouping/reducing. Can you comment on the key distribution? Are most keys
>> unique? Or do you expect lots of keys to be the same in the topic?
>>
>> Thanks
>> Eno
>>
>>
>>> On Mar 10, 2017, at 9:05 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
>>>
>>> Hi All,
>>>
>>> I'm doing a groupBy and reduce on a kstream which results in a state
>> store
>>> being created.
>>>
>>> This state store is growing to be massive, its filled up a 20gb drive.
>> This
>>> feels very unexpected. Is there some cleanup or flushing process for the
>>> state stores that I'm missing or is such a large size expected?
>>>
>>> The topic in question has 96 partitions and the state is about ~200mb
>>> average for each one.
>>>
>>> 175M 1_0
>>> 266M 1_1
>>> 164M 1_10
>>> 177M 1_11
>>> 142M 1_12
>>> 271M 1_13
>>> 158M 1_14
>>> 280M 1_15
>>> 286M 1_16
>>> 181M 1_17
>>> 185M 1_18
>>> 187M 1_19
>>> 281M 1_2
>>> 278M 1_20
>>> 188M 1_21
>>> 262M 1_22
>>> 166M 1_23
>>> 177M 1_24
>>> 268M 1_25
>>> 264M 1_26
>>> 147M 1_27
>>> 179M 1_28
>>> 276M 1_29
>>> 177M 1_3
>>> 157M 1_30
>>> 137M 1_31
>>> 247M 1_32
>>> 275M 1_33
>>> 169M 1_34
>>> 267M 1_35
>>> 283M 1_36
>>> 171M 1_37
>>> 166M 1_38
>>> 277M 1_39
>>> 160M 1_4
>>> 273M 1_40
>>> 278M 1_41
>>> 279M 1_42
>>> 170M 1_43
>>> 139M 1_44
>>> 272M 1_45
>>> 179M 1_46
>>> 283M 1_47
>>> 263M 1_48
>>> 267M 1_49
>>> 181M 1_5
>>> 282M 1_50
>>> 166M 1_51
>>> 161M 1_52
>>> 176M 1_53
>>> 152M 1_54
>>> 172M 1_55
>>> 148M 1_56
>>> 268M 1_57
>>> 144M 1_58
>>> 177M 1_59
>>> 271M 1_6
>>> 279M 1_60
>>> 266M 1_61
>>> 194M 1_62
>>> 177M 1_63
>>> 267M 1_64
>>> 177M 1_65
>>> 271M 1_66
>>> 175M 1_67
>>> 168M 1_68
>>> 140M 1_69
>>> 175M 1_7
>>> 173M 1_70
>>> 179M 1_71
>>> 178M 1_72
>>> 166M 1_73
>>> 180M 1_74
>>> 177M 1_75
>>> 276M 1_76
>>> 177M 1_77
>>> 162M 1_78
>>> 266M 1_79
>>> 194M 1_8
>>> 158M 1_80
>>> 187M 1_81
>>> 162M 1_82
>>> 163M 1_83
>>> 177M 1_84
>>> 286M 1_85
>>> 165M 1_86
>>> 171M 1_87
>>> 162M 1_88
>>> 179M 1_89
>>> 145M 1_9
>>> 166M 1_90
>>> 190M 1_91
>>> 159M 1_92
>>> 284M 1_93
>>> 172M 1_94
>>> 149M 1_95
>>
>>
Re: ~20gb of kafka-streams state, unexpected?
Posted by Ian Duffy <ia...@ianduffy.ie>.
Hi Eno,
Thanks for the fast response.
We are doing a deduplication process here, so yes you are correct the keys
are normally unique. Sounds like a wrong tool for the job issue on my end.
Thanks for your input here.
On 10 March 2017 at 10:59, Eno Thereska <en...@gmail.com> wrote:
> Hi Ian,
>
> Sounds like you have a total topic size of ~20GB (96 partitions x 200mb).
> If most keys are unique then group and reduce might not be as effective in
> grouping/reducing. Can you comment on the key distribution? Are most keys
> unique? Or do you expect lots of keys to be the same in the topic?
>
> Thanks
> Eno
>
>
> > On Mar 10, 2017, at 9:05 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
> >
> > Hi All,
> >
> > I'm doing a groupBy and reduce on a kstream which results in a state
> store
> > being created.
> >
> > This state store is growing to be massive, its filled up a 20gb drive.
> This
> > feels very unexpected. Is there some cleanup or flushing process for the
> > state stores that I'm missing or is such a large size expected?
> >
> > The topic in question has 96 partitions and the state is about ~200mb
> > average for each one.
> >
> > 175M 1_0
> > 266M 1_1
> > 164M 1_10
> > 177M 1_11
> > 142M 1_12
> > 271M 1_13
> > 158M 1_14
> > 280M 1_15
> > 286M 1_16
> > 181M 1_17
> > 185M 1_18
> > 187M 1_19
> > 281M 1_2
> > 278M 1_20
> > 188M 1_21
> > 262M 1_22
> > 166M 1_23
> > 177M 1_24
> > 268M 1_25
> > 264M 1_26
> > 147M 1_27
> > 179M 1_28
> > 276M 1_29
> > 177M 1_3
> > 157M 1_30
> > 137M 1_31
> > 247M 1_32
> > 275M 1_33
> > 169M 1_34
> > 267M 1_35
> > 283M 1_36
> > 171M 1_37
> > 166M 1_38
> > 277M 1_39
> > 160M 1_4
> > 273M 1_40
> > 278M 1_41
> > 279M 1_42
> > 170M 1_43
> > 139M 1_44
> > 272M 1_45
> > 179M 1_46
> > 283M 1_47
> > 263M 1_48
> > 267M 1_49
> > 181M 1_5
> > 282M 1_50
> > 166M 1_51
> > 161M 1_52
> > 176M 1_53
> > 152M 1_54
> > 172M 1_55
> > 148M 1_56
> > 268M 1_57
> > 144M 1_58
> > 177M 1_59
> > 271M 1_6
> > 279M 1_60
> > 266M 1_61
> > 194M 1_62
> > 177M 1_63
> > 267M 1_64
> > 177M 1_65
> > 271M 1_66
> > 175M 1_67
> > 168M 1_68
> > 140M 1_69
> > 175M 1_7
> > 173M 1_70
> > 179M 1_71
> > 178M 1_72
> > 166M 1_73
> > 180M 1_74
> > 177M 1_75
> > 276M 1_76
> > 177M 1_77
> > 162M 1_78
> > 266M 1_79
> > 194M 1_8
> > 158M 1_80
> > 187M 1_81
> > 162M 1_82
> > 163M 1_83
> > 177M 1_84
> > 286M 1_85
> > 165M 1_86
> > 171M 1_87
> > 162M 1_88
> > 179M 1_89
> > 145M 1_9
> > 166M 1_90
> > 190M 1_91
> > 159M 1_92
> > 284M 1_93
> > 172M 1_94
> > 149M 1_95
>
>
Re: ~20gb of kafka-streams state, unexpected?
Posted by Eno Thereska <en...@gmail.com>.
Hi Ian,
Sounds like you have a total topic size of ~20GB (96 partitions x 200mb). If most keys are unique then group and reduce might not be as effective in grouping/reducing. Can you comment on the key distribution? Are most keys unique? Or do you expect lots of keys to be the same in the topic?
Thanks
Eno
> On Mar 10, 2017, at 9:05 AM, Ian Duffy <ia...@ianduffy.ie> wrote:
>
> Hi All,
>
> I'm doing a groupBy and reduce on a kstream which results in a state store
> being created.
>
> This state store is growing to be massive, its filled up a 20gb drive. This
> feels very unexpected. Is there some cleanup or flushing process for the
> state stores that I'm missing or is such a large size expected?
>
> The topic in question has 96 partitions and the state is about ~200mb
> average for each one.
>
> 175M 1_0
> 266M 1_1
> 164M 1_10
> 177M 1_11
> 142M 1_12
> 271M 1_13
> 158M 1_14
> 280M 1_15
> 286M 1_16
> 181M 1_17
> 185M 1_18
> 187M 1_19
> 281M 1_2
> 278M 1_20
> 188M 1_21
> 262M 1_22
> 166M 1_23
> 177M 1_24
> 268M 1_25
> 264M 1_26
> 147M 1_27
> 179M 1_28
> 276M 1_29
> 177M 1_3
> 157M 1_30
> 137M 1_31
> 247M 1_32
> 275M 1_33
> 169M 1_34
> 267M 1_35
> 283M 1_36
> 171M 1_37
> 166M 1_38
> 277M 1_39
> 160M 1_4
> 273M 1_40
> 278M 1_41
> 279M 1_42
> 170M 1_43
> 139M 1_44
> 272M 1_45
> 179M 1_46
> 283M 1_47
> 263M 1_48
> 267M 1_49
> 181M 1_5
> 282M 1_50
> 166M 1_51
> 161M 1_52
> 176M 1_53
> 152M 1_54
> 172M 1_55
> 148M 1_56
> 268M 1_57
> 144M 1_58
> 177M 1_59
> 271M 1_6
> 279M 1_60
> 266M 1_61
> 194M 1_62
> 177M 1_63
> 267M 1_64
> 177M 1_65
> 271M 1_66
> 175M 1_67
> 168M 1_68
> 140M 1_69
> 175M 1_7
> 173M 1_70
> 179M 1_71
> 178M 1_72
> 166M 1_73
> 180M 1_74
> 177M 1_75
> 276M 1_76
> 177M 1_77
> 162M 1_78
> 266M 1_79
> 194M 1_8
> 158M 1_80
> 187M 1_81
> 162M 1_82
> 163M 1_83
> 177M 1_84
> 286M 1_85
> 165M 1_86
> 171M 1_87
> 162M 1_88
> 179M 1_89
> 145M 1_9
> 166M 1_90
> 190M 1_91
> 159M 1_92
> 284M 1_93
> 172M 1_94
> 149M 1_95