You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Susheel Kumar <su...@gmail.com> on 2016/12/20 13:27:15 UTC

Compact Topic - Handling InProcess/Retry/Manual/Completed status

Hello,

I am wondering how we handle the below current workflow in Kafka when using
compact topic.  Do we need to use multiple compact topic's for each of
these directories (InProcess/Retry/Manual/Completed)  and move messages in
between them OR use one topic and application maintains the offset's etc.
Please advise.


   1. All the new orders (json files) are placed under *InProcess*
   directory. If there is an update, and older file hasn't been processed yet,
   the older is file is removed with new one.
   2. The process picks up each json file and submits to Solr. If
   submission successful, json file are moved to *Ingested* directory
   3. If submission fails, it retries one time and moves files to *Ingested*
   folder if successful
   4. If fails, the json file is moved to *Retry* directory
   5. The *Retry* directory files are retried every hour and if they fail,
   they are moved to *Manual* directory for someone to look manually
   6. If successful, Retry files are moved to *Ingested* folder.


Thanks,
Susheel

Re: Compact Topic - Handling InProcess/Retry/Manual/Completed status

Posted by Susheel Kumar <su...@gmail.com>.
Any insight on how we can design below current workflow in Kafka when using
compacted topic OR basically how we handle retry/failures.  Do we need to
use multiple compact topic's for each of these directories
(InProcess/Retry/Manual/Completed)  and move messages in between them OR
use one topic and application maintains the offset's etc.

Current workflow

   1. All the new orders (json files) are placed under *InProcess* directory.
   If there is an update, and older file hasn't been processed yet, the older
   file is removed with new one.
   2. The process picks up each json file and submits to Solr. If
   submission successful, json file are moved to *Ingested* directory
   3. If submission fails, it retries one time and moves files to
*Ingested* folder

   4. If fails, the json file is moved to *Retry* directory
   5. The *Retry* directory files are retried every hour and if they fail,
   they are moved to *Manual* directory for someone to look manually
   6. If successful, Retry files are moved to *Ingested* folder.


Thanks,
Susheel

On Tue, Dec 20, 2016 at 8:27 AM, Susheel Kumar <su...@gmail.com>
wrote:

> Hello,
>
> I am wondering how we handle the below current workflow in Kafka when
> using compact topic.  Do we need to use multiple compact topic's for each
> of these directories (InProcess/Retry/Manual/Completed)  and move
> messages in between them OR use one topic and application maintains the
> offset's etc.  Please advise.
>
>
>    1. All the new orders (json files) are placed under *InProcess*
>    directory. If there is an update, and older file hasn't been processed yet,
>    the older is file is removed with new one.
>    2. The process picks up each json file and submits to Solr. If
>    submission successful, json file are moved to *Ingested* directory
>    3. If submission fails, it retries one time and moves files to
>    *Ingested* folder if successful
>    4. If fails, the json file is moved to *Retry* directory
>    5. The *Retry* directory files are retried every hour and if they
>    fail, they are moved to *Manual* directory for someone to look manually
>    6. If successful, Retry files are moved to *Ingested* folder.
>
>
> Thanks,
> Susheel
>