You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ali Nazemian <al...@gmail.com> on 2018/07/11 13:31:18 UTC
Kafka disk recommendation for production cluster
Hi All,
I was wondering what the disk recommendation is for Kafka cluster? Is it
acceptable to use RAID0 in the case that replication is 3? We are running
on a cloud infrastructure and disk failure is addressed at another level,
so the chance of single disk failure would be very low. Besides, our
version of Kafka does not fully support JBOD as even a single disk failure
can stop a Kafka broker, so we will not get the actual benefit of using
JBOD anyway. However, I am not quite sure how software raid acts in this
situation as there is no option to use HW Raid on cloud.
Regards,
Ali
Re: Kafka disk recommendation for production cluster
Posted by Ali Nazemian <al...@gmail.com>.
I meant because there is no actual Raid Controller for running Raid on
cloud, it is basically Software Based Raid [1].
[1] https://en.wikipedia.org/wiki/RAID#Software-based
On Thu, Jul 12, 2018 at 6:29 PM M. Manna <ma...@gmail.com> wrote:
> From all cases I know so far (including RDBMS storage solution), RAID 10
> has been the choice. Also, the replication you are mentioning is the s/w
> replication nothing to do with RAID 0 setup.
>
>
>
> On 11 July 2018 at 23:59, Ali Nazemian <al...@gmail.com> wrote:
>
> > Thanks. As this proposal is not available for the version of Kafka that
> we
> > are using, we should go with Raid option, but the question is how is it
> > acceptable to use Software Raid0 on a cloud infrastructure?
> >
> > On Thu, 12 Jul. 2018, 08:02 M. Manna, <ma...@gmail.com> wrote:
> >
> > > Dong Lin's KIP -
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 112%3A+Handle+disk+failure+for+JBOD
> > >
> > > Should give you some ideas.
> > >
> > > On 11 July 2018 at 14:31, Ali Nazemian <al...@gmail.com> wrote:
> > >
> > > > Hi All,
> > > >
> > > > I was wondering what the disk recommendation is for Kafka cluster? Is
> > it
> > > > acceptable to use RAID0 in the case that replication is 3? We are
> > running
> > > > on a cloud infrastructure and disk failure is addressed at another
> > level,
> > > > so the chance of single disk failure would be very low. Besides, our
> > > > version of Kafka does not fully support JBOD as even a single disk
> > > failure
> > > > can stop a Kafka broker, so we will not get the actual benefit of
> using
> > > > JBOD anyway. However, I am not quite sure how software raid acts in
> > this
> > > > situation as there is no option to use HW Raid on cloud.
> > > >
> > > > Regards,
> > > > Ali
> > > >
> > >
> >
>
--
A.Nazemian
Re: Kafka disk recommendation for production cluster
Posted by "M. Manna" <ma...@gmail.com>.
From all cases I know so far (including RDBMS storage solution), RAID 10
has been the choice. Also, the replication you are mentioning is the s/w
replication nothing to do with RAID 0 setup.
On 11 July 2018 at 23:59, Ali Nazemian <al...@gmail.com> wrote:
> Thanks. As this proposal is not available for the version of Kafka that we
> are using, we should go with Raid option, but the question is how is it
> acceptable to use Software Raid0 on a cloud infrastructure?
>
> On Thu, 12 Jul. 2018, 08:02 M. Manna, <ma...@gmail.com> wrote:
>
> > Dong Lin's KIP -
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 112%3A+Handle+disk+failure+for+JBOD
> >
> > Should give you some ideas.
> >
> > On 11 July 2018 at 14:31, Ali Nazemian <al...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > I was wondering what the disk recommendation is for Kafka cluster? Is
> it
> > > acceptable to use RAID0 in the case that replication is 3? We are
> running
> > > on a cloud infrastructure and disk failure is addressed at another
> level,
> > > so the chance of single disk failure would be very low. Besides, our
> > > version of Kafka does not fully support JBOD as even a single disk
> > failure
> > > can stop a Kafka broker, so we will not get the actual benefit of using
> > > JBOD anyway. However, I am not quite sure how software raid acts in
> this
> > > situation as there is no option to use HW Raid on cloud.
> > >
> > > Regards,
> > > Ali
> > >
> >
>
Re: Kafka disk recommendation for production cluster
Posted by Ali Nazemian <al...@gmail.com>.
Thanks. As this proposal is not available for the version of Kafka that we
are using, we should go with Raid option, but the question is how is it
acceptable to use Software Raid0 on a cloud infrastructure?
On Thu, 12 Jul. 2018, 08:02 M. Manna, <ma...@gmail.com> wrote:
> Dong Lin's KIP -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-112%3A+Handle+disk+failure+for+JBOD
>
> Should give you some ideas.
>
> On 11 July 2018 at 14:31, Ali Nazemian <al...@gmail.com> wrote:
>
> > Hi All,
> >
> > I was wondering what the disk recommendation is for Kafka cluster? Is it
> > acceptable to use RAID0 in the case that replication is 3? We are running
> > on a cloud infrastructure and disk failure is addressed at another level,
> > so the chance of single disk failure would be very low. Besides, our
> > version of Kafka does not fully support JBOD as even a single disk
> failure
> > can stop a Kafka broker, so we will not get the actual benefit of using
> > JBOD anyway. However, I am not quite sure how software raid acts in this
> > situation as there is no option to use HW Raid on cloud.
> >
> > Regards,
> > Ali
> >
>
Re: Kafka disk recommendation for production cluster
Posted by "M. Manna" <ma...@gmail.com>.
Dong Lin's KIP -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-112%3A+Handle+disk+failure+for+JBOD
Should give you some ideas.
On 11 July 2018 at 14:31, Ali Nazemian <al...@gmail.com> wrote:
> Hi All,
>
> I was wondering what the disk recommendation is for Kafka cluster? Is it
> acceptable to use RAID0 in the case that replication is 3? We are running
> on a cloud infrastructure and disk failure is addressed at another level,
> so the chance of single disk failure would be very low. Besides, our
> version of Kafka does not fully support JBOD as even a single disk failure
> can stop a Kafka broker, so we will not get the actual benefit of using
> JBOD anyway. However, I am not quite sure how software raid acts in this
> situation as there is no option to use HW Raid on cloud.
>
> Regards,
> Ali
>