You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Cristian Makoto Sandiga <cm...@gmail.com> on 2015/05/15 20:34:42 UTC
KafkaSpout with offsets.storage=kafka for HighLoad
Hi,
I want to replace my RedisSpout and i was developing a POC, with
KafkaSpout, but i'm worried for consumer offset manager with zookeeper and
HighLoad write, maybe loose offset, failover, etc.
I was thinking about a zookeeper cluster for any data loose, but i think is
still better save in kafka server offset at least is a kafka
recommendation
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
.
That has any documentation about offset.storage in KafkaSpout i read all
mailing list but i didn't find nothing talking about it, has anybody
working in a possible feature? or maybe somebody has a good recommendation
for me?.
I really will appreciate your help.
Application with normal traffic: 100 000 msgs/seg High 500 000 msgs/seg
replication factor-> 2
1 topic -> 10 partitions
1 topic -> 10 partitions
1 topic -> 5 partition
storm-kafka -> 0.9.4,
kafka_2.10 -> 0.8.2.1
Re: KafkaSpout with offsets.storage=kafka for HighLoad
Posted by Cristian Makoto Sandiga <cm...@gmail.com>.
Thanks , for your answer srikanth.
Current KafkaSpout doesn't support this new feature available in Kafka.
I was looking for something similar a few weeks back.
Srikanth
On Fri, May 15, 2015 at 3:04 PM, Cristian Makoto Sandiga <
cmsandiga@gmail.com> wrote:
> Thank you Jeffery, yes.
>
> 2015-05-15 16:01 GMT-03:00 Jeffery Maass <ma...@gmail.com>:
>
> As I read your email and the link within it, I believe that your question
>> is:
>>
>> * Does there exist or is anyone working on a KafkaSpout which stores its
>> offsets in a highly durable Kafka Topic, instead of within ZooKeeper?
>>
>> Is that a correct interpretation of your question?
>>
>> Thank you for your time!
>>
>> +++++++++++++++++++++
>> Jeff Maass <ma...@gmail.com>
>> linkedin.com/in/jeffmaass
>> stackoverflow.com/users/373418/maassql
>> +++++++++++++++++++++
>>
>>
>> On Fri, May 15, 2015 at 1:34 PM, Cristian Makoto Sandiga <
>> cmsandiga@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I want to replace my RedisSpout and i was developing a POC, with
>>> KafkaSpout, but i'm worried for consumer offset manager with zookeeper and
>>> HighLoad write, maybe loose offset, failover, etc.
>>>
>>> I was thinking about a zookeeper cluster for any data loose, but i think
>>> is still better save in kafka server offset at least is a kafka
>>> recommendation
>>> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
>>> .
>>>
>>> That has any documentation about offset.storage in KafkaSpout i read all
>>> mailing list but i didn't find nothing talking about it, has anybody
>>> working in a possible feature? or maybe somebody has a good recommendation
>>> for me?.
>>>
>>> I really will appreciate your help.
>>>
>>> Application with normal traffic: 100 000 msgs/seg High 500 000 msgs/seg
>>>
>>> replication factor-> 2
>>>
>>> 1 topic -> 10 partitions
>>> 1 topic -> 10 partitions
>>> 1 topic -> 5 partition
>>>
>>> storm-kafka -> 0.9.4,
>>> kafka_2.10 -> 0.8.2.1
>>>
>>
>>
>
Re: KafkaSpout with offsets.storage=kafka for HighLoad
Posted by Srikanth <sr...@gmail.com>.
Current KafkaSpout doesn't support this new feature available in Kafka.
I was looking for something similar a few weeks back.
Srikanth
On Fri, May 15, 2015 at 3:04 PM, Cristian Makoto Sandiga <
cmsandiga@gmail.com> wrote:
> Thank you Jeffery, yes.
>
> 2015-05-15 16:01 GMT-03:00 Jeffery Maass <ma...@gmail.com>:
>
> As I read your email and the link within it, I believe that your question
>> is:
>>
>> * Does there exist or is anyone working on a KafkaSpout which stores its
>> offsets in a highly durable Kafka Topic, instead of within ZooKeeper?
>>
>> Is that a correct interpretation of your question?
>>
>> Thank you for your time!
>>
>> +++++++++++++++++++++
>> Jeff Maass <ma...@gmail.com>
>> linkedin.com/in/jeffmaass
>> stackoverflow.com/users/373418/maassql
>> +++++++++++++++++++++
>>
>>
>> On Fri, May 15, 2015 at 1:34 PM, Cristian Makoto Sandiga <
>> cmsandiga@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I want to replace my RedisSpout and i was developing a POC, with
>>> KafkaSpout, but i'm worried for consumer offset manager with zookeeper and
>>> HighLoad write, maybe loose offset, failover, etc.
>>>
>>> I was thinking about a zookeeper cluster for any data loose, but i think
>>> is still better save in kafka server offset at least is a kafka
>>> recommendation
>>> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
>>> .
>>>
>>> That has any documentation about offset.storage in KafkaSpout i read all
>>> mailing list but i didn't find nothing talking about it, has anybody
>>> working in a possible feature? or maybe somebody has a good recommendation
>>> for me?.
>>>
>>> I really will appreciate your help.
>>>
>>> Application with normal traffic: 100 000 msgs/seg High 500 000 msgs/seg
>>>
>>> replication factor-> 2
>>>
>>> 1 topic -> 10 partitions
>>> 1 topic -> 10 partitions
>>> 1 topic -> 5 partition
>>>
>>> storm-kafka -> 0.9.4,
>>> kafka_2.10 -> 0.8.2.1
>>>
>>
>>
>
Re: KafkaSpout with offsets.storage=kafka for HighLoad
Posted by Cristian Makoto Sandiga <cm...@gmail.com>.
Thank you Jeffery, yes.
2015-05-15 16:01 GMT-03:00 Jeffery Maass <ma...@gmail.com>:
> As I read your email and the link within it, I believe that your question
> is:
>
> * Does there exist or is anyone working on a KafkaSpout which stores its
> offsets in a highly durable Kafka Topic, instead of within ZooKeeper?
>
> Is that a correct interpretation of your question?
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
> On Fri, May 15, 2015 at 1:34 PM, Cristian Makoto Sandiga <
> cmsandiga@gmail.com> wrote:
>
>> Hi,
>>
>> I want to replace my RedisSpout and i was developing a POC, with
>> KafkaSpout, but i'm worried for consumer offset manager with zookeeper and
>> HighLoad write, maybe loose offset, failover, etc.
>>
>> I was thinking about a zookeeper cluster for any data loose, but i think
>> is still better save in kafka server offset at least is a kafka
>> recommendation
>> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
>> .
>>
>> That has any documentation about offset.storage in KafkaSpout i read all
>> mailing list but i didn't find nothing talking about it, has anybody
>> working in a possible feature? or maybe somebody has a good recommendation
>> for me?.
>>
>> I really will appreciate your help.
>>
>> Application with normal traffic: 100 000 msgs/seg High 500 000 msgs/seg
>>
>> replication factor-> 2
>>
>> 1 topic -> 10 partitions
>> 1 topic -> 10 partitions
>> 1 topic -> 5 partition
>>
>> storm-kafka -> 0.9.4,
>> kafka_2.10 -> 0.8.2.1
>>
>
>
Re: KafkaSpout with offsets.storage=kafka for HighLoad
Posted by Jeffery Maass <ma...@gmail.com>.
As I read your email and the link within it, I believe that your question
is:
* Does there exist or is anyone working on a KafkaSpout which stores its
offsets in a highly durable Kafka Topic, instead of within ZooKeeper?
Is that a correct interpretation of your question?
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Fri, May 15, 2015 at 1:34 PM, Cristian Makoto Sandiga <
cmsandiga@gmail.com> wrote:
> Hi,
>
> I want to replace my RedisSpout and i was developing a POC, with
> KafkaSpout, but i'm worried for consumer offset manager with zookeeper and
> HighLoad write, maybe loose offset, failover, etc.
>
> I was thinking about a zookeeper cluster for any data loose, but i think
> is still better save in kafka server offset at least is a kafka
> recommendation
> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
> .
>
> That has any documentation about offset.storage in KafkaSpout i read all
> mailing list but i didn't find nothing talking about it, has anybody
> working in a possible feature? or maybe somebody has a good recommendation
> for me?.
>
> I really will appreciate your help.
>
> Application with normal traffic: 100 000 msgs/seg High 500 000 msgs/seg
>
> replication factor-> 2
>
> 1 topic -> 10 partitions
> 1 topic -> 10 partitions
> 1 topic -> 5 partition
>
> storm-kafka -> 0.9.4,
> kafka_2.10 -> 0.8.2.1
>