You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "arjun4084346 (via GitHub)" <gi...@apache.org> on 2024/07/08 17:52:52 UTC
[PR] add cache to handle insert and delete cdc events coming at a small in… [gobblin]
arjun4084346 opened a new pull request, #3994:
URL: https://github.com/apache/gobblin/pull/3994
…terval for the same dag action
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-XXX
### Description
- [ ] Here are some details about my PR, including screenshots (if applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cache to handle insert and delete cdc events coming at a small in… [gobblin]
Posted by "arjun4084346 (via GitHub)" <gi...@apache.org>.
arjun4084346 commented on code in PR #3994:
URL: https://github.com/apache/gobblin/pull/3994#discussion_r1669265587
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java:
##########
@@ -24,6 +24,8 @@
import java.util.UUID;
import java.util.concurrent.TimeUnit;
+import org.jetbrains.annotations.NotNull;
Review Comment:
This file only has codestyle changes IntelliJ suggested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cache to handle insert and delete cdc events coming at a small in… [gobblin]
Posted by "umustafi (via GitHub)" <gi...@apache.org>.
umustafi commented on code in PR #3994:
URL: https://github.com/apache/gobblin/pull/3994#discussion_r1669433410
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagActionReminderScheduler.java:
##########
@@ -77,28 +95,45 @@ public DagActionReminderScheduler(StdSchedulerFactory schedulerFactory)
* @throws SchedulerException
*/
public void scheduleReminder(DagActionStore.LeaseParams leaseParams, long reminderDurationMillis,
- boolean isDeadlineReminder)
- throws SchedulerException {
+ boolean isDeadlineReminder) throws SchedulerException {
+ DagActionStore.DagAction dagAction = leaseParams.getDagAction();
JobDetail jobDetail = createReminderJobDetail(leaseParams, isDeadlineReminder);
- Trigger trigger = createReminderJobTrigger(leaseParams.getDagAction(), reminderDurationMillis,
+ Trigger trigger = createReminderJobTrigger(dagAction, reminderDurationMillis,
System::currentTimeMillis, isDeadlineReminder);
- log.info("Reminder set for dagAction {} to fire after {} ms, isDeadlineTrigger: {}",
- leaseParams.getDagAction(), reminderDurationMillis, isDeadlineReminder);
- quartzScheduler.scheduleJob(jobDetail, trigger);
+ log.info("Going to set reminder for dagAction {} to fire after {} ms, isDeadlineTrigger: {}",
Review Comment:
let's prefix all of these logs with `DagActionReminderScheduler` so easy to find
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagActionReminderScheduler.java:
##########
@@ -77,28 +95,45 @@ public DagActionReminderScheduler(StdSchedulerFactory schedulerFactory)
* @throws SchedulerException
*/
public void scheduleReminder(DagActionStore.LeaseParams leaseParams, long reminderDurationMillis,
- boolean isDeadlineReminder)
- throws SchedulerException {
+ boolean isDeadlineReminder) throws SchedulerException {
+ DagActionStore.DagAction dagAction = leaseParams.getDagAction();
JobDetail jobDetail = createReminderJobDetail(leaseParams, isDeadlineReminder);
- Trigger trigger = createReminderJobTrigger(leaseParams.getDagAction(), reminderDurationMillis,
+ Trigger trigger = createReminderJobTrigger(dagAction, reminderDurationMillis,
System::currentTimeMillis, isDeadlineReminder);
- log.info("Reminder set for dagAction {} to fire after {} ms, isDeadlineTrigger: {}",
- leaseParams.getDagAction(), reminderDurationMillis, isDeadlineReminder);
- quartzScheduler.scheduleJob(jobDetail, trigger);
+ log.info("Going to set reminder for dagAction {} to fire after {} ms, isDeadlineTrigger: {}",
+ dagAction, reminderDurationMillis, isDeadlineReminder);
+
+ try {
+ if (!this.deleteDeadlineDagActionCache.get(dagAction)) {
+ quartzScheduler.scheduleJob(jobDetail, trigger);
+ } else {
+ log.info("Ignoring {} because the delete equivalent of the same received already.", dagAction);
+ this.deleteDeadlineDagActionCache.put(dagAction, false);
+ }
+ } catch (ObjectAlreadyExistsException e) {
+ log.warn("Reminder job {} already exists in the quartz scheduler. Possibly a duplicate request.", jobDetail.getKey());
Review Comment:
`Reminder job for this dagAction already exists in the DagActionReminderScheduler. Look for duplicate requests for the same action. Job: {}`
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##########
@@ -131,6 +129,7 @@ public static void submitJobToExecutor(DagManagementStateStore dagManagementStat
log.info("Orchestrated job: {} on Executor: {}", DagManagerUtils.getFullyQualifiedJobName(dagNode), specExecutorUri);
dagManagementStateStore.getDagManagerMetrics().incrementJobsSentToExecutor(dagNode);
dagManagementStateStore.addDagNodeState(dagNode, dagId);
+ sendEnforceJobStartDeadlineDagAction(dagManagementStateStore, dagNode);
Review Comment:
comment to explain why moved below to document
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagActionReminderScheduler.java:
##########
@@ -77,28 +95,45 @@ public DagActionReminderScheduler(StdSchedulerFactory schedulerFactory)
* @throws SchedulerException
*/
public void scheduleReminder(DagActionStore.LeaseParams leaseParams, long reminderDurationMillis,
- boolean isDeadlineReminder)
- throws SchedulerException {
+ boolean isDeadlineReminder) throws SchedulerException {
+ DagActionStore.DagAction dagAction = leaseParams.getDagAction();
JobDetail jobDetail = createReminderJobDetail(leaseParams, isDeadlineReminder);
- Trigger trigger = createReminderJobTrigger(leaseParams.getDagAction(), reminderDurationMillis,
+ Trigger trigger = createReminderJobTrigger(dagAction, reminderDurationMillis,
System::currentTimeMillis, isDeadlineReminder);
- log.info("Reminder set for dagAction {} to fire after {} ms, isDeadlineTrigger: {}",
- leaseParams.getDagAction(), reminderDurationMillis, isDeadlineReminder);
- quartzScheduler.scheduleJob(jobDetail, trigger);
+ log.info("Going to set reminder for dagAction {} to fire after {} ms, isDeadlineTrigger: {}",
+ dagAction, reminderDurationMillis, isDeadlineReminder);
+
+ try {
+ if (!this.deleteDeadlineDagActionCache.get(dagAction)) {
+ quartzScheduler.scheduleJob(jobDetail, trigger);
+ } else {
+ log.info("Ignoring {} because the delete equivalent of the same received already.", dagAction);
+ this.deleteDeadlineDagActionCache.put(dagAction, false);
+ }
+ } catch (ObjectAlreadyExistsException e) {
+ log.warn("Reminder job {} already exists in the quartz scheduler. Possibly a duplicate request.", jobDetail.getKey());
+ } catch (JobPersistenceException e) {
+ // this may happen when there is a race condition between this and delete job operation, retry this in that case
+ quartzScheduler.scheduleJob(jobDetail, trigger);
Review Comment:
what is this exception? does the occur if the scheduler attempts to delete it at the same time? clarify comment
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/MysqlDagActionStore.java:
##########
@@ -122,6 +123,9 @@ public void addJobDagAction(String flowGroup, String flowName, long flowExecutio
try {
fillPreparedStatement(flowGroup, flowName, flowExecutionId, jobName, dagActionType, insertStatement);
return insertStatement.executeUpdate();
+ } catch (SQLIntegrityConstraintViolationException e) {
+ log.error(e.getMessage());
Review Comment:
why do we skip throwing an exception here?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManagementTaskStreamImpl.java:
##########
@@ -107,23 +107,10 @@ public DagManagementTaskStreamImpl(Config config, Optional<DagActionStore> dagAc
this.dagManagementStateStore = dagManagementStateStore;
}
- @Override
- public synchronized void addDagAction(DagActionStore.DagAction dagAction) {
- // TODO: Used to track missing dag issue, remove later as needed
- log.info("Add original (non-reminder) dagAction {}", dagAction);
-
- if (!this.leaseParamsQueue.offer(new DagActionStore.LeaseParams(dagAction, false, System.currentTimeMillis()))) {
- throw new RuntimeException(String.format("Could not add dag action to the queue %s", dagAction));
- }
- }
-
- @Override
- public synchronized void addReminderDagAction(DagActionStore.LeaseParams reminderLeaseParams) {
- // TODO: Used to track missing dag issue, remove later as needed
- log.info("Add reminder dagAction {}", reminderLeaseParams);
-
+ public synchronized void addDagAction(DagActionStore.LeaseParams reminderLeaseParams) {
Review Comment:
simply name this lease params since used for both type of actions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cache to handle insert and delete cdc events coming at a small in… [gobblin]
Posted by "umustafi (via GitHub)" <gi...@apache.org>.
umustafi commented on code in PR #3994:
URL: https://github.com/apache/gobblin/pull/3994#discussion_r1669439540
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagActionReminderScheduler.java:
##########
@@ -77,28 +95,45 @@ public DagActionReminderScheduler(StdSchedulerFactory schedulerFactory)
* @throws SchedulerException
*/
public void scheduleReminder(DagActionStore.LeaseParams leaseParams, long reminderDurationMillis,
- boolean isDeadlineReminder)
- throws SchedulerException {
+ boolean isDeadlineReminder) throws SchedulerException {
+ DagActionStore.DagAction dagAction = leaseParams.getDagAction();
JobDetail jobDetail = createReminderJobDetail(leaseParams, isDeadlineReminder);
- Trigger trigger = createReminderJobTrigger(leaseParams.getDagAction(), reminderDurationMillis,
+ Trigger trigger = createReminderJobTrigger(dagAction, reminderDurationMillis,
System::currentTimeMillis, isDeadlineReminder);
- log.info("Reminder set for dagAction {} to fire after {} ms, isDeadlineTrigger: {}",
- leaseParams.getDagAction(), reminderDurationMillis, isDeadlineReminder);
- quartzScheduler.scheduleJob(jobDetail, trigger);
+ log.info("Going to set reminder for dagAction {} to fire after {} ms, isDeadlineTrigger: {}",
+ dagAction, reminderDurationMillis, isDeadlineReminder);
+
+ try {
+ if (!this.deleteDeadlineDagActionCache.get(dagAction)) {
+ quartzScheduler.scheduleJob(jobDetail, trigger);
+ } else {
+ log.info("Ignoring {} because the delete equivalent of the same received already.", dagAction);
Review Comment:
`DagActionReminderScheduler` skipping adding reminder for action because already received delete for this action: {}.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cache to handle insert and delete cdc events coming at a small in… [gobblin]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3994:
URL: https://github.com/apache/gobblin/pull/3994#issuecomment-2215455660
## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/3994?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 38.72%. Comparing base [(`4721d08`)](https://app.codecov.io/gh/apache/gobblin/commit/4721d086f78553cab28e46bf406787df6cc2dea8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`c4a15f6`)](https://app.codecov.io/gh/apache/gobblin/commit/c4a15f60d8524cea729b81c6cc380f17681d2ddf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> :exclamation: There is a different number of reports uploaded between BASE (4721d08) and HEAD (c4a15f6). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (4721d08) | HEAD (c4a15f6) |
>|------|------|------|
>||2|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3994 +/- ##
============================================
- Coverage 45.48% 38.72% -6.76%
+ Complexity 3222 1595 -1627
============================================
Files 705 388 -317
Lines 27558 15957 -11601
Branches 2777 1578 -1199
============================================
- Hits 12535 6180 -6355
+ Misses 13960 9283 -4677
+ Partials 1063 494 -569
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/gobblin/pull/3994?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cache to handle insert and delete cdc events coming at a small in… [gobblin]
Posted by "arjun4084346 (via GitHub)" <gi...@apache.org>.
arjun4084346 closed pull request #3994: add cache to handle insert and delete cdc events coming at a small in…
URL: https://github.com/apache/gobblin/pull/3994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org