Kafka in python

[Moustafa Mohamed Sadik <mo...@natixis.com>]

Hello,

I am using kafka by python packages.

Is it possible for server to retain messages when no consumer is available?

I want to start producer and send message to the server (this part is ok).

After that (when the sending to server is finished), I want to start a consumer and get the messages from the last read message.

Actually when I do that, no message is shown in the consumer or when it is specified in the parameters, I got all messages from beginning.

So how to get the message after the last consumed one when I start my consumer?


Thanks


MOUSTAFA Mohamed Sadik
Apprenti ingénieur en informatique
mohamedsadik.moustafa@natixis.com<ma...@natixis.com>
+33158553880-23880
Natixis SA




C1 - Public Natixis
<http://www.natixis.com/natixis/jcms/rpaz5_78162/fr/natixis>
L'integrite de ce message n'etant pas assuree sur internet, Natixis ne peut etre tenu responsable de son contenu. Toute utilisation ou diffusion non autorisee est interdite. Si vous n'etes pas destinataire de ce message, merci de le detruire et d'avertir l'expediteur. Ensemble, faisons un geste pour l'environnement : n'imprimons nos mails que si necessaire.
Pour consulter l’ensemble des disclaimers de l’entreprise, veuillez consulter l’url : https://www.natixis.com/natixis/upload/docs/application/pdf/2017-09/disclaimer.pdf <https://www.natixis.com/natixis/upload/docs/application/pdf/2017-09/disclaimer.pdf>

The integrity of this message cannot be guaranteed on the Internet. Natixis cannot therefore be considered responsible for the contents. Any unauthorized use or dissemination is
prohibited. If you are not the intended recipient of this message, please delete it and notify the sender. Please respect the environment and print emails only when necessary.
To consult the entire Natixis disclaimers, please use the following link : https://www.natixis.com/natixis/upload/docs/application/pdf/2017-09/disclaimer.pdf <https://www.natixis.com/natixis/upload/docs/application/pdf/2017-09/disclaimer.pdf>

Re: Kafka in python

[Devin Kramer <dk...@rubiconproject.com.INVALID>]

If your consumer is always starting with the same consumer group ID it
should pick up where it left off.   This also assumes your consumer is
correctly sending its offsets to the __consumer_offsets topic and that the
time between it starting and stopping is not so long that whatever
retention you have set on that topic is expiring your consumers offset
value.   That being said Is there a technical reason your consumer can't
simply connect and stay connected and consuming as messages come in rather
then having to start up occasionally?  Although the later should work the
former is probably more inline with a standard kafka consumer, although I
may be speaking above my knowledge grade on that..

On Fri, Jul 12, 2019 at 8:23 AM Moustafa Mohamed Sadik <
mohamedsadik.moustafa@natixis.com> wrote:

> Hello,
>
> I am using kafka by python packages.
>
> Is it possible for server to retain messages when no consumer is available?
>
> I want to start producer and send message to the server (this part is ok).
>
> After that (when the sending to server is finished), I want to start a
> consumer and get the messages from the last read message.
>
> Actually when I do that, no message is shown in the consumer or when it is
> specified in the parameters, I got all messages from beginning.
>
> So how to get the message after the last consumed one when I start my
> consumer?
>
>
> Thanks
>
>
> MOUSTAFA Mohamed Sadik
> Apprenti ingénieur en informatique
> mohamedsadik.moustafa@natixis.com<mailto:mohamedsadik.moustafa@natixis.com
> >
> +33158553880-23880
> Natixis SA
>
>
>
>
> C1 - Public Natixis
> <http://www.natixis.com/natixis/jcms/rpaz5_78162/fr/natixis>
> L'integrite de ce message n'etant pas assuree sur internet, Natixis ne
> peut etre tenu responsable de son contenu. Toute utilisation ou diffusion
> non autorisee est interdite. Si vous n'etes pas destinataire de ce message,
> merci de le detruire et d'avertir l'expediteur. Ensemble, faisons un geste
> pour l'environnement : n'imprimons nos mails que si necessaire.
> Pour consulter l’ensemble des disclaimers de l’entreprise, veuillez
> consulter l’url :
> https://www.natixis.com/natixis/upload/docs/application/pdf/2017-09/disclaimer.pdf
> <
> https://www.natixis.com/natixis/upload/docs/application/pdf/2017-09/disclaimer.pdf
> >
>
> The integrity of this message cannot be guaranteed on the Internet.
> Natixis cannot therefore be considered responsible for the contents. Any
> unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, please
> delete it and notify the sender. Please respect the environment and print
> emails only when necessary.
> To consult the entire Natixis disclaimers, please use the following link :
> https://www.natixis.com/natixis/upload/docs/application/pdf/2017-09/disclaimer.pdf
> <
> https://www.natixis.com/natixis/upload/docs/application/pdf/2017-09/disclaimer.pdf
> >
>


-- 

Devin Kramer Director, Infrastructure Engineering

12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094

RUBICONPROJECT.COM <http://www.rubiconproject.com>  |  JOIN US
<http://rubiconproject.com/join-us/>  |  FOLLOW US
<http://twitter.com/rubiconproject>