You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "William Lo (Jira)" <ji...@apache.org> on 2023/11/16 17:19:00 UTC

[jira] [Created] (GOBBLIN-1959) Introduce Gobblin partial success semantic which fails job on partial success

William Lo created GOBBLIN-1959:
-----------------------------------

             Summary: Introduce Gobblin partial success semantic which fails job on partial success
                 Key: GOBBLIN-1959
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1959
             Project: Apache Gobblin
          Issue Type: Improvement
          Components: gobblin-core
            Reporter: William Lo
            Assignee: Abhishek Tiwari


Currently, the partial success semantics in Gobblin where it follows a commit on task success policy will return a successful job running status even though some tasks fail. This is because in most Gobblin jobs, there would be movement of the watermark after the tasks succeed so we want to follow that semantic to not lead to duplicate ingestion.

However, there are some Gobblin jobs (distcp) which do not follow the watermark semantic, thus a partial success should return a failed status so that the job callers can understand that not every task has succeeded, thus invoke a retry with amended amounts of work.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)