You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "umustafi (via GitHub)" <gi...@apache.org> on 2023/09/13 22:23:53 UTC

[GitHub] [gobblin] umustafi opened a new pull request, #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants

umustafi opened a new pull request, #3771:
URL: https://github.com/apache/gobblin/pull/3771

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [X] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-1907
   
   
   ### Description
   - [X] Here are some details about my PR, including screenshots (if applicable):
   When other participants complete a lease (not just acquire it) in the time that the current participant has attempted the lease, we will see a NULL `lease_acquisition_timestamp` and should appropriately handle the resulting value. This case was missed and we were only expecting another participant to have acquired the lease. 
   We also add warning logs for `JobExceptions` that can be hiding the stack trace of errors encountered in runtime. 
   
   ### Tests
   - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   Tested behavior without change to see NPE and after updates the tests pass without any exceptions 
   
   ### Commits
   - [X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [gobblin] phet commented on a diff in pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #3771:
URL: https://github.com/apache/gobblin/pull/3771#discussion_r1325136912


##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java:
##########
@@ -496,7 +496,9 @@ public void runJob(Properties jobProps, JobListener jobListener) throws JobExcep
           ConfigurationKeys.ORCHESTRATOR_TRIGGER_EVENT_TIME_NEVER_SET_VAL);
       this.orchestrator.orchestrate(flowSpec, jobProps, Long.parseLong(triggerTimestampMillis));
     } catch (Exception e) {
-      throw new JobException("Failed to run Spec: " + jobProps.getProperty(ConfigurationKeys.JOB_NAME_KEY), e);
+      String exceptionPrefix = "Failed to run Spec: " + jobProps.getProperty(ConfigurationKeys.JOB_NAME_KEY);
+      log.warn(exceptionPrefix + " with stacktrace", e);

Review Comment:
   nit: "with stacktrace" is fine, but "because" might read better



##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/MysqlMultiActiveLeaseArbiter.java:
##########
@@ -544,6 +553,7 @@ static class GetEventInfoResult {
   */
   @Data
   static class SelectInfoResult {
+    public static final long LEASE_COMPLETED_VALUE = -1;

Review Comment:
   how about an `Optional` instead?



##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/api/MysqlMultiActiveLeaseArbiterTest.java:
##########
@@ -17,8 +17,10 @@
 
 package org.apache.gobblin.runtime.api;
 
+import com.google.common.base.Optional;

Review Comment:
   let's use java `Optional`
   
   (seems the original class under test mistakenly used the deprecated guava one... I presume it's not too difficult at this early stage to correct)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [gobblin] codecov-commenter commented on pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3771:
URL: https://github.com/apache/gobblin/pull/3771#issuecomment-1718460073

   ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/3771?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#3771](https://app.codecov.io/gh/apache/gobblin/pull/3771?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (22c6687) into [master](https://app.codecov.io/gh/apache/gobblin/commit/d6f546e91f16d72d66c12182d86b416d54a8db83?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d6f546e) will **decrease** coverage by `3.74%`.
   > The diff coverage is `n/a`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3771      +/-   ##
   ============================================
   - Coverage     49.74%   46.01%   -3.74%     
   + Complexity     9438     2177    -7261     
   ============================================
     Files          1767      416    -1351     
     Lines         69294    17981   -51313     
     Branches       7888     2189    -5699     
   ============================================
   - Hits          34471     8274   -26197     
   + Misses        31636     8828   -22808     
   + Partials       3187      879    -2308     
   ```
   
   
   [see 1355 files with indirect coverage changes](https://app.codecov.io/gh/apache/gobblin/pull/3771/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [gobblin] umustafi commented on a diff in pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants

Posted by "umustafi (via GitHub)" <gi...@apache.org>.
umustafi commented on code in PR #3771:
URL: https://github.com/apache/gobblin/pull/3771#discussion_r1325143516


##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/api/MysqlMultiActiveLeaseArbiterTest.java:
##########
@@ -17,8 +17,10 @@
 
 package org.apache.gobblin.runtime.api;
 
+import com.google.common.base.Optional;

Review Comment:
   The guava optional is actually used throughout the code in `FlowTriggerHandler, GobblinServiceJobScheduler, LeaseArbiter, `test class etc...I'd rather not spend as much time refactoring this right now since a lot of these pieces of code interact with one another so switching between `Java/Guava` optionals will get confusing and I'd need to change it in a lot of places. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [gobblin] umustafi commented on a diff in pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants

Posted by "umustafi (via GitHub)" <gi...@apache.org>.
umustafi commented on code in PR #3771:
URL: https://github.com/apache/gobblin/pull/3771#discussion_r1325143516


##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/api/MysqlMultiActiveLeaseArbiterTest.java:
##########
@@ -17,8 +17,10 @@
 
 package org.apache.gobblin.runtime.api;
 
+import com.google.common.base.Optional;

Review Comment:
   I replaced the guava optional with the Java one in the `MysqlMultiActiveLeaseArbiter[Test]` but am leaving the other related classes as is for now. Will fix going forward.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [gobblin] ZihanLi58 merged pull request #3771: [GOBBLIN-1907] Handle Lease Completion From Other Multi-active Participants

Posted by "ZihanLi58 (via GitHub)" <gi...@apache.org>.
ZihanLi58 merged PR #3771:
URL: https://github.com/apache/gobblin/pull/3771


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org