You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "William Lo (Jira)" <ji...@apache.org> on 2023/11/16 17:19:00 UTC
[jira] [Created] (GOBBLIN-1959) Introduce Gobblin partial success semantic which fails job on partial success
William Lo created GOBBLIN-1959:
-----------------------------------
Summary: Introduce Gobblin partial success semantic which fails job on partial success
Key: GOBBLIN-1959
URL: https://issues.apache.org/jira/browse/GOBBLIN-1959
Project: Apache Gobblin
Issue Type: Improvement
Components: gobblin-core
Reporter: William Lo
Assignee: Abhishek Tiwari
Currently, the partial success semantics in Gobblin where it follows a commit on task success policy will return a successful job running status even though some tasks fail. This is because in most Gobblin jobs, there would be movement of the watermark after the tasks succeed so we want to follow that semantic to not lead to duplicate ingestion.
However, there are some Gobblin jobs (distcp) which do not follow the watermark semantic, thus a partial success should return a failed status so that the job callers can understand that not every task has succeeded, thus invoke a retry with amended amounts of work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)