You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "umustafi (via GitHub)" <gi...@apache.org> on 2023/09/13 22:23:53 UTC
[GitHub] [gobblin] umustafi opened a new pull request, #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants
umustafi opened a new pull request, #3771:
URL: https://github.com/apache/gobblin/pull/3771
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [X] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1907
### Description
- [X] Here are some details about my PR, including screenshots (if applicable):
When other participants complete a lease (not just acquire it) in the time that the current participant has attempted the lease, we will see a NULL `lease_acquisition_timestamp` and should appropriately handle the resulting value. This case was missed and we were only expecting another participant to have acquired the lease.
We also add warning logs for `JobExceptions` that can be hiding the stack trace of errors encountered in runtime.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
Tested behavior without change to see NPE and after updates the tests pass without any exceptions
### Commits
- [X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants
Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #3771:
URL: https://github.com/apache/gobblin/pull/3771#discussion_r1325136912
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java:
##########
@@ -496,7 +496,9 @@ public void runJob(Properties jobProps, JobListener jobListener) throws JobExcep
ConfigurationKeys.ORCHESTRATOR_TRIGGER_EVENT_TIME_NEVER_SET_VAL);
this.orchestrator.orchestrate(flowSpec, jobProps, Long.parseLong(triggerTimestampMillis));
} catch (Exception e) {
- throw new JobException("Failed to run Spec: " + jobProps.getProperty(ConfigurationKeys.JOB_NAME_KEY), e);
+ String exceptionPrefix = "Failed to run Spec: " + jobProps.getProperty(ConfigurationKeys.JOB_NAME_KEY);
+ log.warn(exceptionPrefix + " with stacktrace", e);
Review Comment:
nit: "with stacktrace" is fine, but "because" might read better
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/MysqlMultiActiveLeaseArbiter.java:
##########
@@ -544,6 +553,7 @@ static class GetEventInfoResult {
*/
@Data
static class SelectInfoResult {
+ public static final long LEASE_COMPLETED_VALUE = -1;
Review Comment:
how about an `Optional` instead?
##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/api/MysqlMultiActiveLeaseArbiterTest.java:
##########
@@ -17,8 +17,10 @@
package org.apache.gobblin.runtime.api;
+import com.google.common.base.Optional;
Review Comment:
let's use java `Optional`
(seems the original class under test mistakenly used the deprecated guava one... I presume it's not too difficult at this early stage to correct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter commented on pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3771:
URL: https://github.com/apache/gobblin/pull/3771#issuecomment-1718460073
## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/3771?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#3771](https://app.codecov.io/gh/apache/gobblin/pull/3771?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (22c6687) into [master](https://app.codecov.io/gh/apache/gobblin/commit/d6f546e91f16d72d66c12182d86b416d54a8db83?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d6f546e) will **decrease** coverage by `3.74%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #3771 +/- ##
============================================
- Coverage 49.74% 46.01% -3.74%
+ Complexity 9438 2177 -7261
============================================
Files 1767 416 -1351
Lines 69294 17981 -51313
Branches 7888 2189 -5699
============================================
- Hits 34471 8274 -26197
+ Misses 31636 8828 -22808
+ Partials 3187 879 -2308
```
[see 1355 files with indirect coverage changes](https://app.codecov.io/gh/apache/gobblin/pull/3771/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a diff in pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants
Posted by "umustafi (via GitHub)" <gi...@apache.org>.
umustafi commented on code in PR #3771:
URL: https://github.com/apache/gobblin/pull/3771#discussion_r1325143516
##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/api/MysqlMultiActiveLeaseArbiterTest.java:
##########
@@ -17,8 +17,10 @@
package org.apache.gobblin.runtime.api;
+import com.google.common.base.Optional;
Review Comment:
The guava optional is actually used throughout the code in `FlowTriggerHandler, GobblinServiceJobScheduler, LeaseArbiter, `test class etc...I'd rather not spend as much time refactoring this right now since a lot of these pieces of code interact with one another so switching between `Java/Guava` optionals will get confusing and I'd need to change it in a lot of places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a diff in pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants
Posted by "umustafi (via GitHub)" <gi...@apache.org>.
umustafi commented on code in PR #3771:
URL: https://github.com/apache/gobblin/pull/3771#discussion_r1325143516
##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/api/MysqlMultiActiveLeaseArbiterTest.java:
##########
@@ -17,8 +17,10 @@
package org.apache.gobblin.runtime.api;
+import com.google.common.base.Optional;
Review Comment:
I replaced the guava optional with the Java one in the `MysqlMultiActiveLeaseArbiter[Test]` but am leaving the other related classes as is for now. Will fix going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] ZihanLi58 merged pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants
Posted by "ZihanLi58 (via GitHub)" <gi...@apache.org>.
ZihanLi58 merged PR #3771:
URL: https://github.com/apache/gobblin/pull/3771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org