You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Graeme Wallace <gr...@farecompare.com> on 2013/10/03 01:35:50 UTC

Performance of jBOD

We notice that we get a lot higher performance from Kafka up until the
point where disk reads are taking place. (which is to be expected as its
reading from filesystem cache).

However, given that we have 12x7200rpm SATA disks configured as jBOD - what
sort of disk read performance should i see reported by dstat when things
are running flat out ?

I'm seeing 250-340Mb sustained.

-- 
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018

Re: Performance of jBOD

Posted by Neha Narkhede <ne...@gmail.com>.
The average observed per broker write bandwidth is roughly 5 MB/s and read
bandwidth is roughly 20 MB/s. But
these statistics are from clusters that are no where close to maxing out
the write/read bandwidth available. In most cases though, you will hit the
network bottleneck (~125 MB/s) first.

Thanks,
Neha


On Thu, Oct 3, 2013 at 11:08 AM, Graeme Wallace <
graeme.wallace@farecompare.com> wrote:

> Jun,
>
> Just to follow up - have you any stats on what disk read + write throughput
> you are getting in production at LinkedIn ?
>
> regards,
>
> Graeme
>
>
>
> On Thu, Oct 3, 2013 at 7:17 AM, Graeme Wallace <
> graeme.wallace@farecompare.com> wrote:
>
> > Its whatever dstat reports - i think it must be MBytes ?
> >
> >
> > On Wed, Oct 2, 2013 at 10:29 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> >> If you use a reasonable fetch size in the consumer (e.g. 100+KB) or
> above,
> >> you could probably get most of the sequential scan performance from
> those
> >> disks. Do you mean 250 Mbits or MBytes?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Wed, Oct 2, 2013 at 4:35 PM, Graeme Wallace <
> >> graeme.wallace@farecompare.com> wrote:
> >>
> >> > We notice that we get a lot higher performance from Kafka up until the
> >> > point where disk reads are taking place. (which is to be expected as
> its
> >> > reading from filesystem cache).
> >> >
> >> > However, given that we have 12x7200rpm SATA disks configured as jBOD -
> >> what
> >> > sort of disk read performance should i see reported by dstat when
> things
> >> > are running flat out ?
> >> >
> >> > I'm seeing 250-340Mb sustained.
> >> >
> >> > --
> >> > Graeme Wallace
> >> > CTO
> >> > FareCompare.com
> >> > O: 972 588 1414
> >> > M: 214 681 9018
> >> >
> >>
> >
> >
> >
> > --
> > Graeme Wallace
> > CTO
> > FareCompare.com
> > O: 972 588 1414
> > M: 214 681 9018
> >
> >
>
>
> --
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018
>

Re: Performance of jBOD

Posted by Graeme Wallace <gr...@farecompare.com>.
Jun,

Just to follow up - have you any stats on what disk read + write throughput
you are getting in production at LinkedIn ?

regards,

Graeme



On Thu, Oct 3, 2013 at 7:17 AM, Graeme Wallace <
graeme.wallace@farecompare.com> wrote:

> Its whatever dstat reports - i think it must be MBytes ?
>
>
> On Wed, Oct 2, 2013 at 10:29 PM, Jun Rao <ju...@gmail.com> wrote:
>
>> If you use a reasonable fetch size in the consumer (e.g. 100+KB) or above,
>> you could probably get most of the sequential scan performance from those
>> disks. Do you mean 250 Mbits or MBytes?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Wed, Oct 2, 2013 at 4:35 PM, Graeme Wallace <
>> graeme.wallace@farecompare.com> wrote:
>>
>> > We notice that we get a lot higher performance from Kafka up until the
>> > point where disk reads are taking place. (which is to be expected as its
>> > reading from filesystem cache).
>> >
>> > However, given that we have 12x7200rpm SATA disks configured as jBOD -
>> what
>> > sort of disk read performance should i see reported by dstat when things
>> > are running flat out ?
>> >
>> > I'm seeing 250-340Mb sustained.
>> >
>> > --
>> > Graeme Wallace
>> > CTO
>> > FareCompare.com
>> > O: 972 588 1414
>> > M: 214 681 9018
>> >
>>
>
>
>
> --
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018
>
>


-- 
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018

Re: Performance of jBOD

Posted by Graeme Wallace <gr...@farecompare.com>.
Its whatever dstat reports - i think it must be MBytes ?


On Wed, Oct 2, 2013 at 10:29 PM, Jun Rao <ju...@gmail.com> wrote:

> If you use a reasonable fetch size in the consumer (e.g. 100+KB) or above,
> you could probably get most of the sequential scan performance from those
> disks. Do you mean 250 Mbits or MBytes?
>
> Thanks,
>
> Jun
>
>
> On Wed, Oct 2, 2013 at 4:35 PM, Graeme Wallace <
> graeme.wallace@farecompare.com> wrote:
>
> > We notice that we get a lot higher performance from Kafka up until the
> > point where disk reads are taking place. (which is to be expected as its
> > reading from filesystem cache).
> >
> > However, given that we have 12x7200rpm SATA disks configured as jBOD -
> what
> > sort of disk read performance should i see reported by dstat when things
> > are running flat out ?
> >
> > I'm seeing 250-340Mb sustained.
> >
> > --
> > Graeme Wallace
> > CTO
> > FareCompare.com
> > O: 972 588 1414
> > M: 214 681 9018
> >
>



-- 
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018

Re: Performance of jBOD

Posted by Jun Rao <ju...@gmail.com>.
If you use a reasonable fetch size in the consumer (e.g. 100+KB) or above,
you could probably get most of the sequential scan performance from those
disks. Do you mean 250 Mbits or MBytes?

Thanks,

Jun


On Wed, Oct 2, 2013 at 4:35 PM, Graeme Wallace <
graeme.wallace@farecompare.com> wrote:

> We notice that we get a lot higher performance from Kafka up until the
> point where disk reads are taking place. (which is to be expected as its
> reading from filesystem cache).
>
> However, given that we have 12x7200rpm SATA disks configured as jBOD - what
> sort of disk read performance should i see reported by dstat when things
> are running flat out ?
>
> I'm seeing 250-340Mb sustained.
>
> --
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018
>