You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "adithyachakilam (via GitHub)" <gi...@apache.org> on 2024/03/05 06:00:46 UTC
[PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
adithyachakilam opened a new pull request, #16042:
URL: https://github.com/apache/druid/pull/16042
### Description
Fixed the bug where completion task reports are not being generated on `index_parallel` tasks.
The fix made in #15981 wrongly assumes that every time an index task is not run as stand alone task just skip the reports. But it is not true. An index task could be run inside a compaction task or a batch ingestion task but we do need reports to be published by the index task if it is run as part of batch ingestion task.
Also, this PR tries to address the comments on #15981 made after merged.
<hr>
##### Key changed/added classes in this PR
* `CompactionTask`
* `IndexTask`
* `ParallelIndexSupervisorTask`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [x] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "adithyachakilam (via GitHub)" <gi...@apache.org>.
adithyachakilam commented on code in PR #16042:
URL: https://github.com/apache/druid/pull/16042#discussion_r1513293231
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java:
##########
@@ -580,9 +586,11 @@ public TaskStatus runTask(final TaskToolbox toolbox)
}
}
- private void writeCompletionReports(TaskToolbox toolbox)
+
+ private void updateAndWriteCompletionReports(TaskToolbox toolbox)
{
- if (isStandAloneTask) {
+ completionReports = getTaskCompletionReports();
+ if (shouldSendReports) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 commented on code in PR #16042:
URL: https://github.com/apache/druid/pull/16042#discussion_r1513282969
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java:
##########
@@ -580,9 +586,11 @@ public TaskStatus runTask(final TaskToolbox toolbox)
}
}
- private void writeCompletionReports(TaskToolbox toolbox)
+
+ private void updateAndWriteCompletionReports(TaskToolbox toolbox)
{
- if (isStandAloneTask) {
+ completionReports = getTaskCompletionReports();
+ if (shouldSendReports) {
Review Comment:
how come we decided to call getTaskCompletionReports() again here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "adithyachakilam (via GitHub)" <gi...@apache.org>.
adithyachakilam commented on PR #16042:
URL: https://github.com/apache/druid/pull/16042#issuecomment-1979416049
@georgew5656, That actually makes sense, fixed it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #16042:
URL: https://github.com/apache/druid/pull/16042#issuecomment-1979025051
Thanks for the quick fix, @adithyachakilam . I have left a few comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16042:
URL: https://github.com/apache/druid/pull/16042#discussion_r1513736815
##########
indexing-service/src/test/java/org/apache/druid/indexing/common/task/TaskReportSerdeTest.java:
##########
@@ -94,6 +94,52 @@ public void testSerde() throws Exception
Assert.assertEquals(reportMap1, reportMap2);
}
+ @Test
+ public void testSerializationOnMissingPartitionStats() throws Exception
Review Comment:
Thanks for adding the test.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java:
##########
@@ -217,6 +215,11 @@ public IndexTask(
);
}
+ /**
+ * @param isStandAloneTask used to specify if indextask.run() is run as a part of another task
+ * skips writing reports and cleanup if not a standalone task
+ */
+
Review Comment:
Nit: extra line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 commented on code in PR #16042:
URL: https://github.com/apache/druid/pull/16042#discussion_r1513281957
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java:
##########
@@ -580,9 +586,11 @@ public TaskStatus runTask(final TaskToolbox toolbox)
}
}
- private void writeCompletionReports(TaskToolbox toolbox)
+
+ private void updateAndWriteCompletionReports(TaskToolbox toolbox)
{
- if (isStandAloneTask) {
+ completionReports = getTaskCompletionReports();
+ if (shouldSendReports) {
Review Comment:
shouldWriteReports maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 commented on code in PR #16042:
URL: https://github.com/apache/druid/pull/16042#discussion_r1513282969
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java:
##########
@@ -580,9 +586,11 @@ public TaskStatus runTask(final TaskToolbox toolbox)
}
}
- private void writeCompletionReports(TaskToolbox toolbox)
+
+ private void updateAndWriteCompletionReports(TaskToolbox toolbox)
{
- if (isStandAloneTask) {
+ completionReports = getTaskCompletionReports();
+ if (shouldSendReports) {
Review Comment:
how come we decided to call getTaskCompletionReports() again here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16042:
URL: https://github.com/apache/druid/pull/16042#discussion_r1513015091
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java:
##########
@@ -226,7 +226,8 @@ public IndexTask(
IndexIngestionSpec ingestionSchema,
Map<String, Object> context,
int maxAllowedLockCount,
- boolean isStandAloneTask
+ boolean shouldCleanup,
+ boolean shouldSendReports
Review Comment:
Javadoc for these arguments would be helpful.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java:
##########
@@ -1261,8 +1252,13 @@ private Map<String, TaskReport> getTaskCompletionReports(TaskStatus taskStatus,
);
}
- private void writeCompletionReports(TaskToolbox toolbox)
+ private void updateAndWriteCompletionReports(
+ TaskToolbox toolbox,
+ TaskStatus status,
+ boolean segmentAvailabilityConfirmationCompleted
Review Comment:
This is a member field of this class and need not be passed as an argument.
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java:
##########
@@ -203,7 +203,7 @@ public class ParallelIndexSupervisorTask extends AbstractBatchIndexTask implemen
private IngestionState ingestionState;
private Map<String, TaskReport> completionReports;
- private final Boolean isCompactionTask;
+ private final boolean isCompactionTask;
Review Comment:
The corresponding argument in the constructor should be changed too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 commented on PR #16042:
URL: https://github.com/apache/druid/pull/16042#issuecomment-1979380105
> is this actually the behavior we want? it seems to contradict the behavior of other situations where we run a task inside another task where we expect the parent task to write its own reports
looking at the code it seems to me all the other branches in ParallelIndexSupervisorTask call writeCompletionReports using the task report of the subtask generated (runRangePartitionMultiPhaseParallel, runSinglePhaseParallel, runHashPartitionMultiPhaseParallel)
I see there's already this code
```
if (currentSubTaskHolder.setTask(sequentialIndexTask)
&& sequentialIndexTask.isReady(toolbox.getTaskActionClient())) {
TaskStatus status = sequentialIndexTask.run(toolbox);
completionReports = sequentialIndexTask.getCompletionReports();
return status;
} else {
String msg = "Task was asked to stop. Finish as failed";
LOG.info(msg);
return TaskStatus.failure(getId(), msg);
}
```
shouldn't we just call writeCompletionReports here instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 merged PR #16042:
URL: https://github.com/apache/druid/pull/16042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "georgew5656 (via GitHub)" <gi...@apache.org>.
georgew5656 commented on PR #16042:
URL: https://github.com/apache/druid/pull/16042#issuecomment-1979372591
is this actually the behavior we want? it seems to contradict the behavior of other situations where we run a task inside another task where we expect the parent task to write its own reports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fixup!: #15981 Missing completion reports on index_parallel tasks (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #16042:
URL: https://github.com/apache/druid/pull/16042#discussion_r1513352917
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java:
##########
@@ -1261,7 +1252,16 @@
);
}
- private void writeCompletionReports(TaskToolbox toolbox)
+ private void updateAndWriteCompletionReports(
+ TaskToolbox toolbox,
Review Comment:
## Useless parameter
The parameter 'toolbox' is never used.
[Show more details](https://github.com/apache/druid/security/code-scanning/6981)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org