You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dominique Rondé <do...@codecentric.de> on 2016/03/17 13:39:49 UTC
How to start with the first Kafka Message
Hi folks,
i have a kafka topic with messages from the last 7 days. Now i have a
new flink streaming process and like to consume the messages from the
beginning. If I just bring up the topology, the consumer starts from
this moment and not from beginning.
THX
Dominique
Re: How to start with the first Kafka Message
Posted by Dominique Rondé <do...@codecentric.de>.
I just place the parameterTool.getProperties() in the
FlinkKafkaConsumer082 constructor. Now I am able to decide with
--auto.offset.reset smallest at the commandline each time i start up a
route.
Thanks for your hints!
Dominique
Am 18.03.2016 um 03:02 schrieb Balaji Rajagopalan:
> If it is a one time thing, you could run the kafka consumer script
> which has the --from-beginning option, and redirect the output as a
> socketstream and consume in flink.
>
> On Thu, Mar 17, 2016 at 7:24 PM, Till Rohrmann <trohrmann@apache.org
> <ma...@apache.org>> wrote:
>
> Hi Dominique,
>
> have you tried setting the Kafka property
> |props.put("auto.offset.reset", "smallest");|?
>
> Cheers,
> Till
>
>
>
> On Thu, Mar 17, 2016 at 1:39 PM, Dominique Rondé
> <dominique.ronde@codecentric.de
> <ma...@codecentric.de>> wrote:
>
> Hi folks,
>
> i have a kafka topic with messages from the last 7 days. Now i
> have a new flink streaming process and like to consume the
> messages from the beginning. If I just bring up the topology,
> the consumer starts from this moment and not from beginning.
>
> THX
>
> Dominique
>
>
>
--
Dominique Rondé | Senior Consultant
codecentric AG | Kreuznacherstrasse 30 | 60486 Frankfurt | Deutschland
mobil: +49 (0) 172.7182592
www.codecentric.de | blog.codecentric.de | www.meettheexperts.de | www.more4fi.de
Sitz der Gesellschaft: Solingen | HRB 25917| Amtsgericht Wuppertal
Vorstand: Michael Hochgürtel . Mirko Novakovic . Rainer Vehns
Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen Schütz
Re: How to start with the first Kafka Message
Posted by Balaji Rajagopalan <ba...@olacabs.com>.
If it is a one time thing, you could run the kafka consumer script which
has the --from-beginning option, and redirect the output as a socketstream
and consume in flink.
On Thu, Mar 17, 2016 at 7:24 PM, Till Rohrmann <tr...@apache.org> wrote:
> Hi Dominique,
>
> have you tried setting the Kafka property props.put("auto.offset.reset",
> "smallest");?
>
> Cheers,
> Till
>
>
> On Thu, Mar 17, 2016 at 1:39 PM, Dominique Rondé <
> dominique.ronde@codecentric.de> wrote:
>
>> Hi folks,
>>
>> i have a kafka topic with messages from the last 7 days. Now i have a new
>> flink streaming process and like to consume the messages from the
>> beginning. If I just bring up the topology, the consumer starts from this
>> moment and not from beginning.
>>
>> THX
>>
>> Dominique
>>
>
>
Re: How to start with the first Kafka Message
Posted by Till Rohrmann <tr...@apache.org>.
Hi Dominique,
have you tried setting the Kafka property props.put("auto.offset.reset",
"smallest");?
Cheers,
Till
On Thu, Mar 17, 2016 at 1:39 PM, Dominique Rondé <
dominique.ronde@codecentric.de> wrote:
> Hi folks,
>
> i have a kafka topic with messages from the last 7 days. Now i have a new
> flink streaming process and like to consume the messages from the
> beginning. If I just bring up the topology, the consumer starts from this
> moment and not from beginning.
>
> THX
>
> Dominique
>