You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Hanghang Liu (Jira)" <ji...@apache.org> on 2023/11/02 04:27:00 UTC

[jira] [Closed] (GOBBLIN-1740) Split Workunit for Kafka Streaming jobs

     [ https://issues.apache.org/jira/browse/GOBBLIN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hanghang Liu closed GOBBLIN-1740.
---------------------------------
    Resolution: Duplicate

> Split Workunit for Kafka Streaming jobs
> ---------------------------------------
>
>                 Key: GOBBLIN-1740
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1740
>             Project: Apache Gobblin
>          Issue Type: Improvement
>          Components: gobblin-kafka
>            Reporter: Hanghang Liu
>            Assignee: Shirshanka Das
>            Priority: Major
>
> When adding task from a running Helix streaming job, we need the capability to split an existing workunit. 
> We can reuse most of the attributes within the old workunit, and just need to update following props:
> task.id
> writer.output.dir
> partition.id
> gobblin.kafka.streaming.numPartitions
> Since we get the partition watermark from state store, so we don't need to re-calculate that within the workunit. All kafka related properties can also be reused.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)