RE: starting of at a small scale, single ec2 instance with 7.5 GB RAM with kafka

[S Ahmed <sa...@gmail.com>]

Hi,

I like how kafka operates, but I'm wondering if it is possible to run
everything on a single ec2 instance with 7.5 GB RAM.

So that would be zookeeper and a single kafka broker.

I would have a separate server to consume from the broker.

Producers would be from my web servers.


I don't want to complicate things as i don't really need failover or
redundancy etc.  I just want to keep things simple.

I'll have a single topic, and a few partitions because I want the guarantee
that the messages are in order.


Is this something that would be really out of the norm and not recommended?
i.e. nobody really uses it this way and who knows what is going to happen?
:)

Re: starting of at a small scale, single ec2 instance with 7.5 GB RAM with kafka

[Niek Sanders <ni...@gmail.com>]

If you really only care about small scale (no HA, no horizontal
scaling), you could also consider using Redis instead of Kafka for
queueing.

- Niek


On Tue, May 20, 2014 at 2:23 PM, S Ahmed <sa...@gmail.com> wrote:
> Yes agreed, but I have done some load testing before and kafka was doing
> 10's of thousands of messages per second.
>
> If I am doing only hundreds, I think it could handle it for now.  Like I
> said this is small scale.
>
>
> On Tue, May 20, 2014 at 2:51 PM, Neha Narkhede <ne...@gmail.com>wrote:
>
>> It is not recommended to install both kafka and zookeeper on the same box
>> as both would fight for the available memory and performance will degrade.
>>
>> Thanks
>> Neha
>>
>>
>> On Mon, May 19, 2014 at 7:29 AM, S Ahmed <sa...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I like how kafka operates, but I'm wondering if it is possible to run
>> > everything on a single ec2 instance with 7.5 GB RAM.
>> >
>> > So that would be zookeeper and a single kafka broker.
>> >
>> > I would have a separate server to consume from the broker.
>> >
>> > Producers would be from my web servers.
>> >
>> >
>> > I don't want to complicate things as i don't really need failover or
>> > redundancy etc.  I just want to keep things simple.
>> >
>> > I'll have a single topic, and a few partitions because I want the
>> guarantee
>> > that the messages are in order.
>> >
>> >
>> > Is this something that would be really out of the norm and not
>> recommended?
>> > i.e. nobody really uses it this way and who knows what is going to
>> happen?
>> > :)
>> >
>>

Re: starting of at a small scale, single ec2 instance with 7.5 GB RAM with kafka

[S Ahmed <sa...@gmail.com>]

Yes agreed, but I have done some load testing before and kafka was doing
10's of thousands of messages per second.

If I am doing only hundreds, I think it could handle it for now.  Like I
said this is small scale.


On Tue, May 20, 2014 at 2:51 PM, Neha Narkhede <ne...@gmail.com>wrote:

> It is not recommended to install both kafka and zookeeper on the same box
> as both would fight for the available memory and performance will degrade.
>
> Thanks
> Neha
>
>
> On Mon, May 19, 2014 at 7:29 AM, S Ahmed <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > I like how kafka operates, but I'm wondering if it is possible to run
> > everything on a single ec2 instance with 7.5 GB RAM.
> >
> > So that would be zookeeper and a single kafka broker.
> >
> > I would have a separate server to consume from the broker.
> >
> > Producers would be from my web servers.
> >
> >
> > I don't want to complicate things as i don't really need failover or
> > redundancy etc.  I just want to keep things simple.
> >
> > I'll have a single topic, and a few partitions because I want the
> guarantee
> > that the messages are in order.
> >
> >
> > Is this something that would be really out of the norm and not
> recommended?
> > i.e. nobody really uses it this way and who knows what is going to
> happen?
> > :)
> >
>

Re: starting of at a small scale, single ec2 instance with 7.5 GB RAM with kafka

[Neha Narkhede <ne...@gmail.com>]

It is not recommended to install both kafka and zookeeper on the same box
as both would fight for the available memory and performance will degrade.

Thanks
Neha


On Mon, May 19, 2014 at 7:29 AM, S Ahmed <sa...@gmail.com> wrote:

> Hi,
>
> I like how kafka operates, but I'm wondering if it is possible to run
> everything on a single ec2 instance with 7.5 GB RAM.
>
> So that would be zookeeper and a single kafka broker.
>
> I would have a separate server to consume from the broker.
>
> Producers would be from my web servers.
>
>
> I don't want to complicate things as i don't really need failover or
> redundancy etc.  I just want to keep things simple.
>
> I'll have a single topic, and a few partitions because I want the guarantee
> that the messages are in order.
>
>
> Is this something that would be really out of the norm and not recommended?
> i.e. nobody really uses it this way and who knows what is going to happen?
> :)
>