You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "homatthew (via GitHub)" <gi...@apache.org> on 2023/04/12 20:01:27 UTC

[GitHub] [gobblin] homatthew opened a new pull request, #3677: [GOBBLIN-1813] Helix workflows submission timeouts are configurable

homatthew opened a new pull request, #3677:
URL: https://github.com/apache/gobblin/pull/3677

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

### JIRA
- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-1813] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1813

### Description
- [ ] Here are some details about my PR, including screenshots (if applicable):

#### Problem Statement:
When Helix is under tremendous load, Helix workflows can easily exceed the 5 minute timeout. When the timeout is exceeded, our jobs clean up the Gobblin job state. Gobblin considers these workflows effectively dead.

If Helix were to schedule these workflows, the workflows will spin up and the tasks assigned will immediately have a `FileNotFoundException`, causing an immediate termination of the container. This ironically puts a greater load on ZK / Helix and can further increase the delays in future workflow submissions.

#### Solution:
The workflow timeouts need to be not hardcoded so that users can adjust the timeout based on their workflow needs and helix / zk cluster size.

This is just a single change as part of a series of changes for improving how Gobblin uses Helix. There are other issues that need to be addressed like:
- Offline helix instance purging should occur before workflow submission
- Helix lost node fencing should occur on each new ApplicationMaster, since Application masters do not hand over containers when an AM dies and is replaced
- Helix workflows should have a configurable safeguard for how often a workflow can be resubmitted / replaced in order to prevent multiple external systems from doing unsafe concurrent mutations to a running workflow. A replanner should not occur too frequently because the work is redundant and the time between workflows is not sufficient for anything to change.

#### Caveats:
This code is shared between GaaS and Kafka ETL code. The default timeout remains unchanged, and is backwards compatible.

### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:

### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
3. Subject is limited to 50 characters
4. Subject does not end with a period
5. Subject uses the imperative mood ("add", not "adding")
6. Body wraps at 72 characters
7. Body explains "what" and "why", not "how"

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org