KafkaSpout

[siddharth ubale <si...@gmail.com>]

Hi all,

I am using a kafka spout for reading data from a producer into my storm
topology. On the kafkaconfig :
1. When i set Kafka.forcefromstart = true .. it read from the beginning ,
however if i close my topology midway and run the topology again by setting
Kafka.forcefromstart = false , it does not read from the last read position
of the partition i.e the place from where i stopped the topology.
2. I also tried using the newly provided values like

 kafkaConfig1.startOffsetTime = kafka.api.OffsetRequest.EarliestTime()
(-1,-2,-3 also)

kafkaConfig1.maxOffsetBehind =100;

but i could not get the kafka spout to read as mentioned.

Can someone please explain how i could use the above features as the storm
github page says the above setting are self explanatory and no explanation
is provided. Can someone pls share their idea of reading from offsets?


Thanks,
Siddharth

Re: KafkaSpout

[siddharth ubale <si...@gmail.com>]

Hi harsha,

Do you mean that o.9.2 incubating has those changes??
i am using the latest release of storm i.e 0.9.2 incubating...

i am unable to pass on :
public static LocalCluster lc = new LocalCluster(localhost,2181);

please do let me know..


Thanks,
Sidharth Ubale

On Tue, Sep 16, 2014 at 7:21 PM, Harsha <ma...@harsha.io> wrote:

> Are you running this using storm LocalCluster and you want to use
> external zookeeper for LocalCluster?.
> The latest code has those changes you can pass  params to
> LocalCluster(localhost,2181).
> -Harsha
>
> On Mon, Sep 15, 2014, at 10:30 PM, siddharth ubale wrote:
> > Hi harsha,
> >
> > Yes i did check with the above mentioned spoutconfig setup....
> > I think i need to bypass the in memory zookeeper which i am unable to do
> > in
> > local mode of storm. Any thoughts on that?
> >
> > Thanks,
> > Siddharth ubale
>

Re: KafkaSpout

[Harsha <ma...@harsha.io>]

Are you running this using storm LocalCluster and you want to use
external zookeeper for LocalCluster?.
The latest code has those changes you can pass  params to
LocalCluster(localhost,2181). 
-Harsha

On Mon, Sep 15, 2014, at 10:30 PM, siddharth ubale wrote:
> Hi harsha,
> 
> Yes i did check with the above mentioned spoutconfig setup....
> I think i need to bypass the in memory zookeeper which i am unable to do
> in
> local mode of storm. Any thoughts on that?
> 
> Thanks,
> Siddharth ubale

Re: KafkaSpout

[siddharth ubale <si...@gmail.com>]

Hi harsha,

Yes i did check with the above mentioned spoutconfig setup....
I think i need to bypass the in memory zookeeper which i am unable to do in
local mode of storm. Any thoughts on that?

Thanks,
Siddharth ubale

Re: KafkaSpout

[Harsha <ka...@harsha.io>]

Siddharth,
        Did you try using SpoutConfig
spoutConfig.forceFromStart = false
spoutConfig.startOffsetTime = kafka.api.OffsetRequest.LatestTime()

There are some good examples here on integrating storm and kafka

https://github.com/miguno/kafka-storm-starter
https://github.com/miguno/kafka-storm-starter/blob/develop/src/main/scala/com/miguno/kafkastorm/storm/KafkaStormDemo.scala

-Harsha
On Mon, Sep 15, 2014, at 04:37 AM, siddharth ubale wrote:
> Hi all,
> 
> I am using a kafka spout for reading data from a producer into my storm
> topology. On the kafkaconfig :
> 1. When i set Kafka.forcefromstart = true .. it read from the beginning ,
> however if i close my topology midway and run the topology again by
> setting
> Kafka.forcefromstart = false , it does not read from the last read
> position
> of the partition i.e the place from where i stopped the topology.
> 2. I also tried using the newly provided values like
> 
>  kafkaConfig1.startOffsetTime = kafka.api.OffsetRequest.EarliestTime()
> (-1,-2,-3 also)
> 
> kafkaConfig1.maxOffsetBehind =100;
> 
> but i could not get the kafka spout to read as mentioned.
> 
> Can someone please explain how i could use the above features as the
> storm
> github page says the above setting are self explanatory and no
> explanation
> is provided. Can someone pls share their idea of reading from offsets?
> 
> 
> Thanks,
> Siddharth