You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/12/14 16:05:25 UTC

[GitHub] [druid] capistrant opened a new pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

capistrant opened a new pull request #10676:
URL: https://github.com/apache/druid/pull/10676


   <!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
   
   <!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
   
   <!-- If you are a committer, follow the PR action item checklist for committers:
   https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
   
   ### Description
   
   <!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
   
   <!-- Describe your patch: what did you change in code? How did you fix the problem? -->
   
   <!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
   
   #### High Level Description
   
   Add configuration in `tuningConfig` for end user to tell Indexing Service to wait for segments to become available for query before completing the indexing task. The configuration is a timeout value in milliseconds to prevent waiting forever. If the timeout expires, the task still succeeds, but the task reports will indicate that Druid was not able to confirm that the segments became available for query.
   
   This new configuration stems from my experience operating a production cluster with many tenants who often have the same complaint: "My indexing job is complete but the latest data is not available right when the job finishes". This addresses that by letting the client set a reasonable timeout. After the job completes, they can parse the ingestion report and see if their segments became available. More often than not, with a reasonable timeout, their segments will indeed be available right when the job completes.
   
   #### Implementation
   
   A lot of the code is already written for realtime handoffs. I extracted that code out of the realtime packages into a Java package so it is less confusing as to why non-realtime tasks are using it. `org.apache.druid.segment.handoff` is a new package in `druid-server` module.
   
   `AbstractBatchIndexTask` gets a new method, `waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)` that handles the waiting. Batch Indexing implementations leverage this method at the end of their ingestion task code if the client's tuningConfig has a non-zero wait time for segment availability. Default is to not wait.
   
   A new key:value pair is added to the IngestionStatsAndErrorsTaskReport `segmentAvailabilityConfirmed`. This is a boolean that indicates if the job was able to confirm query availability of the new segments before finishing. Parallel index task supervisor did not previously have this report, so this PR adds the report with the needed availability key:value pair so all of simple native, parallel native, and hadoop native can implement this availability wait.
   
   #### Alternatives
   
   https://github.com/apache/druid/releases#19-datasource-loadstatus became available in druid 0.20.0. However, I worry about giving ingestion clients the green light to hit this API endpoint due to the possible expense of the calls depending on the questions asked.
   
   <!--
   In each section, please describe design decisions made, including:
    - Choice of algorithms
    - Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
    - Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
    - Method organization and design (how the logic is split between methods, parameters and return types)
    - Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
   -->
   
   
   <!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
   
   <!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
   
   <!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
   
   <hr>
   
   This PR has:
   - [ ] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist above are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * AbstractBatchIndexTask
    * HadoopIndexTask
    * IndexTask
    * ParallelIndexSupervisorTask
    * HadoopTuningConfig
    * ParallelIndexTuningConfig
    * IngestionStatsAndErrorsTaskReportData
    * AbstractITBatchIndexTest
    * ITHadoopIndexTest
    * ITBestEffortRollupParallelIndexTest
    * ITIndexerTest
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r556922336



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AppenderatorDriverRealtimeIndexTask.java
##########
@@ -584,7 +584,8 @@ protected boolean isFirehoseDrainableByClosing(FirehoseFactory firehoseFactory)
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                false

Review comment:
       I think this should be errorMsg == null. also add javadoc 
   
   /**
      * Return a map of reports for the task.
      *
      * A successfull task should always have a null errorMsg. A falied task should always have a non-null
      * errorMsg. Nullable error message for the task. null if task succeeded.
      *
      * @return Map of reports for the task.
      */




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] lgtm-com[bot] commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-744586871


   This pull request **introduces 1 alert** when merging 027938ea80e385136b32c405f8046b9cef675fc1 into 64f97e7003ebc922e404ff49a31969993669747c - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-228dcd6147cb7770cb451d256ebc1948856ac909)
   
   **new alerts:**
   
   * 1 for Dereferenced variable may be null


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r552220064



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/IngestionStatsAndErrorsTaskReportData.java
##########
@@ -41,17 +41,22 @@
   @Nullable
   private String errorMsg;
 
+  @JsonProperty
+  private boolean segmentAvailabilityConfirmed;
+
   public IngestionStatsAndErrorsTaskReportData(
       @JsonProperty("ingestionState") IngestionState ingestionState,
       @JsonProperty("unparseableEvents") Map<String, Object> unparseableEvents,
       @JsonProperty("rowStats") Map<String, Object> rowStats,
-      @JsonProperty("errorMsg") @Nullable String errorMsg
+      @JsonProperty("errorMsg") @Nullable String errorMsg,
+      @JsonProperty("segmentAvailabilityConfirmed") boolean segmentAvailabilityConfirmed

Review comment:
       my company deploys a large multi-tenant cluster with a services layer for ingestion that our tenants use. these tenants don't just want to know when their task succeeds, they also want to know when data from batch ingest is available for querying. This solution allows us to prevent the ingestion services layer and/or individual tenants from banging on Druid APIs trying to see if their data is available after ingestion.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r604464490



##########
File path: integration-tests/src/test/java/org/apache/druid/tests/indexer/AbstractITBatchIndexTest.java
##########
@@ -280,6 +334,18 @@ private void submitTaskAndWait(
       );
     }
 
+    if (segmentAvailabilityConfirmationPair.lhs != null && segmentAvailabilityConfirmationPair.lhs) {
+      TaskReport reportRaw = indexer.getTaskReport(taskID).get("ingestionStatsAndErrors");
+      IngestionStatsAndErrorsTaskReport report = (IngestionStatsAndErrorsTaskReport) reportRaw;
+      IngestionStatsAndErrorsTaskReportData reportData = (IngestionStatsAndErrorsTaskReportData) report.getPayload();
+      if (segmentAvailabilityConfirmationPair.rhs != null) {
+        Assert.assertEquals(
+            Boolean.valueOf(reportData.isSegmentAvailabilityConfirmed()),
+            segmentAvailabilityConfirmationPair.rhs
+        );
+      }
+    }
+

Review comment:
       hmm, my gut is that no we don't want to remove that without some (potentially large) surgery to the indexing IT suite. I added on my own logic for checking the ingestion report for handoff status. But that is separate from what other tests are using this loop for. I would bet that the two ideas could be merged together more gracefully by replacing waitForSegmentsToLoad by using the new handoff config and waiting for the task to complete and then checking the report and failing if the report doesn't indicate the segments were loaded. Not sure if it is in scope for this PR, especially after we revoked support for the handoff for compaction tasks (unless there are no Compaction IT out there that use waitForSegmentsToLoad=true, I'd have to check).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r561152125



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/TaskSerdeTest.java
##########
@@ -268,7 +268,8 @@ public void testIndexTaskSerde() throws Exception
                 null,
                 null,
                 null,
-                null
+                null,

Review comment:
       I think these need asserts




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603527199



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {
+      log.warn("Asked to wait for availability for <= 0 seconds?! Requested waitTimeout: [%s]", waitTimeout);
+      return false;
+    }
+    log.info("Waiting for [%d] segments to be loaded by the cluster...", segmentsToWaitFor.size());
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+    ExecutorService exec = Execs.directExecutor();
+    CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.debug(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            doneSignal.countDown();
+          }
+      );
+    }
+
+    try {
+      return doneSignal.await(waitTimeout, TimeUnit.MILLISECONDS);
+    }
+    catch (InterruptedException e) {
+      log.warn("Interrupted while waiting for segment availablity; Unable to confirm availability!");
+      Thread.currentThread().interrupt();
+      return false;
+    }
+    finally {
+      notifier.close();

Review comment:
       I guess the main question is would be in regards to that existing try/catch. Should that catch be extracted to the try with resource block instead of being nested?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant edited a comment on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant edited a comment on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-811464591


   @suneet-s I took a stab at a CompactionTuningConfig that basically fronts the ParallelIndexTuningConfig that it had been using. This allows us to enforce a check that a non-null or 0 value for `awaitSegmentAvailabilityTimeoutMillis` and causes the job to fail if set differently in the provided tuningConfig. Since this is all the TuningConfig does, I suggest we leave it undocumented and add documentation stating `awaitSegmentAvailabilityTimeoutMillis` is not supported for compaction tasks at this time. 
   
   I still have to test this new code locally, but I wanted to get it up for review and CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r552224901



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.warn("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;

Review comment:
       hmm, I suppose you are right. I guess I initially thought that since I didn't confirm any segments were loaded, I report back that segments were not confirmed to be loaded. However, that false value would be more confusing to an end user of the report compared to returning true... so true seems like right way to go




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601880700



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {
+      log.warn("Asked to wait for availability for <= 0 seconds?! Requested waitTimeout: [%s]", waitTimeout);
+      return false;
+    }
+    log.info("Waiting for [%d] segments to be loaded by the cluster...", segmentsToWaitFor.size());
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+    ExecutorService exec = Execs.directExecutor();
+    CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(

Review comment:
       I suppose it is possible. But it should not matter. CoordinatorBasedSegmentHandoffNotifier should handle that gracefully. We add the callbacks and that notifier is periodically iterating over them and checking for handoff completion which is just an api call to the coordinator. So if I create callback for segment A after it was handed off, the notifier should just complete the callback successfully the next time it does the handoff checks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601884079



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       I think we might. I don't think I was writing this with the intention of having compaction tasks support this. It really wouldn't add value since compaction isn't materially changing the underlying data. But I guess since compaction is spinning up a parallel indexing task, we kinda need to consider it so I will have to check on this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603501367



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       Yes, that's what I was imagining. That being said, I'm open to other options if you had something in mind.
   
   I'm basing this on the principle that if I (the user) do something that Druid doesn't expect, Druid tells me I'm doing something unexpected, and failing fast is better than failing in some obscure way later on in the process.
   
   A clear error message on the failed job like "`awaitSegmentAvailabilityTimeoutMillis ` is not supported for compaction. Please remove this from the spec and re-submit the compaction job." would tell users exactly how to fix the failed job - hence, the nice UX.
   
   A future PR for this support with auto-compaction / compaction would be super cool though.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-807704561


   > Reviewed about 80 files. Posting my incomplete review. I don't think I've found anything major, just a couple of small things and a question about how the SegmentHandoffNotifier works
   
   I appreciate you starting to dig through all of this. I pushed up some of the more trivial changes from comments and addressed the others as well. I think you have made a good point about compaction. I never wrote this intending for compaction to use the handoff, but since it seems I inadvertently allow its support, I should have some kind of testing. I'm looking into that now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601886172



##########
File path: integration-tests/src/test/java/org/apache/druid/tests/coordinator/duty/ITAutoCompactionTest.java
##########
@@ -376,6 +376,11 @@ private void loadData(String indexTask) throws Exception
   {
     String taskSpec = getResourceAsString(indexTask);
     taskSpec = StringUtils.replace(taskSpec, "%%DATASOURCE%%", fullDatasourceName);
+    taskSpec = StringUtils.replace(
+        taskSpec,
+        "%%SEGMENT_AVAIL_TIMEOUT_MILLIS%%",
+        jsonMapper.writeValueAsString("0")

Review comment:
       ya so this relates back to your comment on compaction above. It seems that the smart thing to do is add testing since it is supported by default despite my not intentionally supporting it




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r564942716



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/IngestionStatsAndErrorsTaskReportData.java
##########
@@ -41,17 +41,22 @@
   @Nullable
   private String errorMsg;
 
+  @JsonProperty
+  private boolean segmentAvailabilityConfirmed;
+
   public IngestionStatsAndErrorsTaskReportData(
       @JsonProperty("ingestionState") IngestionState ingestionState,
       @JsonProperty("unparseableEvents") Map<String, Object> unparseableEvents,
       @JsonProperty("rowStats") Map<String, Object> rowStats,
-      @JsonProperty("errorMsg") @Nullable String errorMsg
+      @JsonProperty("errorMsg") @Nullable String errorMsg,
+      @JsonProperty("segmentAvailabilityConfirmed") boolean segmentAvailabilityConfirmed

Review comment:
       > my company deploys a large multi-tenant cluster with a services layer for ingestion that our tenants use. these tenants don't just want to know when their task succeeds, they also want to know when data from batch ingest is available for querying. This solution allows us to prevent the ingestion services layer and/or individual tenants from banging on Druid APIs trying to see if their data is available after ingestion.
   
   I understand this, but my question is more like what people expect when segment handoff fails. In streaming ingestion, the handoff failure causes task failure (this behavior seems arguable, but that's what it does now) and thus people's expectation is that they could see some data dropped after handoff failures until new tasks read the same data and publishes the same segments again. However, since there is no realtime querying in batch ingestion, I don't think tasks should fail on handoff failures (which is what this PR does! :slightly_smiling_face:), but then what will be people's expectation? Are they going to be just OK with handoff failures and wait indefinitely until historicals load new segments (the current behavior)? Do they want to know why the handoff failed? Do they want to know how long it took before the handoff failed? These questions are not clear to me.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r544091878



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java
##########
@@ -491,6 +497,31 @@ private boolean isParallelMode()
     return isParallelMode(baseInputSource, ingestionSchema.getTuningConfig());
   }
 
+  /**
+   * Attempt to wait for indexed segments to become available on the cluster.
+   * @param reportsMap Map containing information with published segments that we are going to wait for.
+   */
+  private void waitForSegmentAvailability(Map<String, PushedSegmentsReport> reportsMap)
+  {
+    ArrayList<DataSegment> segmentsToWaitFor = new ArrayList<>();
+    reportsMap.values()
+              .forEach(report -> {
+                segmentsToWaitFor.addAll(report.getNewSegments());
+              });
+    ExecutorService availabilityExec = Execs.singleThreaded("ParallelTaskAvailabilityWaitExec");
+    try {
+      segmentAvailabilityConfirmationCompleted = waitForSegmentAvailability(
+          toolbox,
+          availabilityExec,
+          segmentsToWaitFor,
+          ingestionSchema.getTuningConfig().getAwaitSegmentAvailabilityTimeoutMillis()

Review comment:
       nit: Could we extract `ingestionSchema.getTuningConfig().getAwaitSegmentAvailabilityTimeoutMillis()` at class level if possible?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r565391843



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/IngestionStatsAndErrorsTaskReportData.java
##########
@@ -41,17 +41,22 @@
   @Nullable
   private String errorMsg;
 
+  @JsonProperty
+  private boolean segmentAvailabilityConfirmed;
+
   public IngestionStatsAndErrorsTaskReportData(
       @JsonProperty("ingestionState") IngestionState ingestionState,
       @JsonProperty("unparseableEvents") Map<String, Object> unparseableEvents,
       @JsonProperty("rowStats") Map<String, Object> rowStats,
-      @JsonProperty("errorMsg") @Nullable String errorMsg
+      @JsonProperty("errorMsg") @Nullable String errorMsg,
+      @JsonProperty("segmentAvailabilityConfirmed") boolean segmentAvailabilityConfirmed

Review comment:
       > > my company deploys a large multi-tenant cluster with a services layer for ingestion that our tenants use. these tenants don't just want to know when their task succeeds, they also want to know when data from batch ingest is available for querying. This solution allows us to prevent the ingestion services layer and/or individual tenants from banging on Druid APIs trying to see if their data is available after ingestion.
   > 
   > I understand this, but my question is more like what people expect when segment handoff fails. In streaming ingestion, the handoff failure causes task failure (this behavior seems arguable, but that's what it does now) and thus people's expectation is that they could see some data dropped after handoff failures until new tasks read the same data and publishes the same segments again. However, since there is no realtime querying in batch ingestion, I don't think tasks should fail on handoff failures (which is what this PR does! 🙂), but then what will be people's expectation? Are they going to be just OK with handoff failures and wait indefinitely until historicals load new segments (the current behavior)? Do they want to know why the handoff failed? Do they want to know how long it took before the handoff failed? These questions are not clear to me.
   
   good question. for my specific case the service that end users interact with really wanted to be able to answer this question for the end user:
   * Is the data that I ingested in this job completely loaded for querying?
   
   For us a simple yes/no will suffice. The cluster operators would have the goal of having 100% of jobs successfully handoff data before the timeout, but when that doesn't happen our users simply want to know that they may need to wait longer. We are simply trying to be transparent and report the point in time status. The onus of finding out when the data is fully loaded if this timeout expired before loading, would fall on a different solution (TBD).
   
   You're right, we intentionally did not fail these tasks because Historical nodes loading the segments is detached from whether or not the data was written to deepstore/metastore (if that failed the task should and likely would fail due to existing code paths). We don't want our end users thinking they need to re-run their jobs when this is much more likely to be an issue of the coordinator not having assigned servers to load segments by the time the timeout expired.
   
   Why the handoff failed would be something I as an operator am more interested compared to a user (unless that user is also an operator). I think that would be very difficult to communicate in these reports since the indexing task doesn't know much about what the rest of the cluster is doing.
   
   Knowing how long it took before the time out could be found in the spec, but I guess it could be useful to add that value to the report as well if you think users would want to have quick reference. I think that rather than having that static value, it could be cool to have the dynamic time waited for handoff. Maybe it is the static value because we hit the timeout. but as an operator I would enjoy seeing how long each successful job waited for handoff. what do you think about that?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r544091714



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()

Review comment:
       nit: Each time we call the `waitForSegmentAvailability` function will communicate with coordinator no matter what the value of `waitTimeout` is and bring extra pressure to the Coordiantor, because there may hundreds of thousands of batch tasks per day. 
   Maybe we can do a **double check** here, like when `waitTimeout<=0` then skip all the `waitForSegmentAvailability ` if possible , just in case that call `waitForSegmentAvailability` function without checking `waitTimeout`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601879349



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");

Review comment:
       Well it will happen in a case when a job doesn't actually ingest any data. It isn't necessarily a bad thing, but it is something I wanted to make note of. I'm going to remove the `!?` to make it seem less important. it really is just informative.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601885386



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/IndexTaskSerdeTest.java
##########
@@ -267,6 +272,7 @@ public void testBestEffortRollupWithHashedPartitionsSpec()
         true,
         10,
         100,
+        null,
         null

Review comment:
       there was not any new explicit test written the other tests in that file do use -1L and 1L as test values. I just updated another of the existing tests to use 0L as well.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r544181990



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,

Review comment:
       Nit: Since the runnable is a simple call without blocking or computation, we could just pass `Execs.directExecutor` here. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] lgtm-com[bot] commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-744870421


   This pull request **introduces 1 alert** when merging 8b9d26d07527b8b2b6b1a8ed09dc0ac36e353797 into 0ad27c06da4e0bb195a67ed623cc8fbba5967e27 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-57a96773a0a587e3ec39d38f407caf73431abbc4)
   
   **new alerts:**
   
   * 1 for Dereferenced variable may be null


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r552223417



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
##########
@@ -1058,7 +1058,8 @@ private synchronized void persistSequences() throws IOException
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                false

Review comment:
       yes you are right.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-769432417


   > Cool, are you working on "the different solution"? That would be interesting too.
   
   I have not thought too much about it yet. but if our desired solution results in new druid features it would definitely be contributed back upstream
   
   > regarding adding to the report with a "timeWaitedForSegmentAvailability" type key value pair
   
   I think we definitely want this so I will be working on follow on PR soon
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r607434654



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
##########
@@ -146,7 +151,7 @@
   @Nullable
   private final ClientCompactionTaskGranularitySpec granularitySpec;
   @Nullable
-  private final ParallelIndexTuningConfig tuningConfig;
+  private final CompactionTuningConfig tuningConfig;

Review comment:
       I think this change means that compaction jobs that are submitted with a `ParallelIndexTuningConfig` (using type = `index_parallel`) will start failing after this change - is this correct?
   
   If so, I think instead of introducing this as a breaking change, we can just add the Precondition check that you have in the `CompactionTuningConfig`  into the constructor for the compaction task.
   
   If this isn't a breaking change, I like that there's a separate tuningConfig for compaction tasks, so that in the future, this config can be more easily optimized.
   
   
   cc @maytasm Since I've seen you make some improvements around compaction recently




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r561152125



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/TaskSerdeTest.java
##########
@@ -268,7 +268,8 @@ public void testIndexTaskSerde() throws Exception
                 null,
                 null,
                 null,
-                null
+                null,

Review comment:
       I think this needs an assert




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r544180581



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.info(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            synchronized (availabilityCondition) {
+              segmentsToWaitFor.remove(s);

Review comment:
       not sure but removing items from the collection while iterating it may cause an error. Instead, we could use a counter and a completableFuture together. Something like this 
   ```
   CompletableFuture<Void> uberFuture = CompletableFuture.completedFuture(null);
   for (DataSegment s : segmentsToWaitFor) {
         CompletableFuture<Void> future = new CompletableFuture<>();
         notifier.registerSegmentHandoffCallback(
             new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
             exec,
             () -> {
               log.info(
                   "Confirmed availability for [%s]. Removing from list of segments to wait for",
                   s.getId()
               );
              future.complete(null);
            }
         );
        uberFuture = uberFuture.thenCombine(future, (a, b) -> null);
       }
   uberFuture.get(waitTimeout, TimeUnit.MILLISECONDS)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-811464591


   @suneet-s I took a stab at a CompactionTuningConfig that basically fronts the ParallelIndexTuningConfig that it had been using. This allows us to enforce a check that a non-null or 0 value for `awaitSegmentAvailabilityTimeoutMillis` causes the job to fail. Since this is all the TuningConfig does, I suggest we leave it undocumented and add documentation stating `awaitSegmentAvailabilityTimeoutMillis` is not supported for compaction tasks at this time. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603508123



##########
File path: integration-tests/src/test/java/org/apache/druid/tests/indexer/AbstractITBatchIndexTest.java
##########
@@ -280,6 +334,18 @@ private void submitTaskAndWait(
       );
     }
 
+    if (segmentAvailabilityConfirmationPair.lhs != null && segmentAvailabilityConfirmationPair.lhs) {
+      TaskReport reportRaw = indexer.getTaskReport(taskID).get("ingestionStatsAndErrors");
+      IngestionStatsAndErrorsTaskReport report = (IngestionStatsAndErrorsTaskReport) reportRaw;
+      IngestionStatsAndErrorsTaskReportData reportData = (IngestionStatsAndErrorsTaskReportData) report.getPayload();
+      if (segmentAvailabilityConfirmationPair.rhs != null) {
+        Assert.assertEquals(
+            Boolean.valueOf(reportData.isSegmentAvailabilityConfirmed()),
+            segmentAvailabilityConfirmationPair.rhs
+        );
+      }
+    }
+

Review comment:
       Do we want to remove the retry loop in here https://github.com/apache/druid/pull/10676/files#diff-6dbfd938d89c6c209d36efc181277afa535629677dfe6b5a1710190c2f9d8ee3L308-L311
   
   if `IngestionStatsAndErrorsTaskReport` say that the segment Availability was confirmed. If it isn't true on the first attempt, that means the task report was wrong - am I understanding this correctly?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603470274



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       Given this discussion, can we add a guardrail that says if someone tries to issue a compaction task when `awaitSegmentAvailabilityTimeoutMillis` is non null and != 0 that this is not supported. I think that's the easiest way forward to get the PR merged and it's a nice UX because it tells users that what they are doing is untested. Similar comment for auto-compaction (and other ingestion tasks that I may have missed)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601890945



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/IngestionStatsAndErrorsTaskReportData.java
##########
@@ -41,17 +41,22 @@
   @Nullable
   private String errorMsg;
 
+  @JsonProperty
+  private boolean segmentAvailabilityConfirmed;
+
   public IngestionStatsAndErrorsTaskReportData(
       @JsonProperty("ingestionState") IngestionState ingestionState,
       @JsonProperty("unparseableEvents") Map<String, Object> unparseableEvents,
       @JsonProperty("rowStats") Map<String, Object> rowStats,
-      @JsonProperty("errorMsg") @Nullable String errorMsg
+      @JsonProperty("errorMsg") @Nullable String errorMsg,
+      @JsonProperty("segmentAvailabilityConfirmed") boolean segmentAvailabilityConfirmed

Review comment:
       So these will be created/written by individual indexing tasks. then stored wherever the cluster stores task logs. And I believe the only way they are ever accessed by Druid is streamed from their location directly to an API caller without ever deserializing them. So I don't think there is any possibility for issues during an upgrade here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603491793



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       > Given this discussion, can we add a guardrail that says if someone tries to issue a compaction task when `awaitSegmentAvailabilityTimeoutMillis` is non null and != 0 that this is not supported. I think that's the easiest way forward to get the PR merged and it's a nice UX because it tells users that what they are doing is untested. Similar comment for auto-compaction (and other ingestion tasks that I may have missed)
   
   Are you expecting a job submit failure then?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603509988



##########
File path: integration-tests/src/test/java/org/apache/druid/tests/indexer/ITCombiningFirehoseFactoryIndexTest.java
##########
@@ -59,13 +61,28 @@ public void testIndexData() throws Exception
           throw new RuntimeException(e);
         }
       };
+      final Function<String, String> transform = spec -> {
+        try {
+          return StringUtils.replace(
+              spec,
+              "%%SEGMENT_AVAIL_TIMEOUT_MILLIS%%",
+              jsonMapper.writeValueAsString("0")

Review comment:
       Some comments on this pattern since I've seen it across a few tests
   - Does this change mean we lose coverage on a missing `SEGMENT_AVAIL_TIMEOUT_MILLIS`? Since this is the default mode, it would be good to have coverage for that missing change
   - Do we have / want tests where this is set to a high enough number that we don't need a re-try loop in the integration tests while waiting for the segments to be available




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-745473962


   @capistrant thanks for the PR. This idea seems useful. I will take a look soon.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zhangyue19921010 commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-745888065


   @capistrant Nice idea. Looking forward to being merged! 👍 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r564931175



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(

Review comment:
       Good point. I agree. I think it would be better to not do such refactoring for parallel or hadoop task in this PR. But it would be still nice to reuse the same logic in both streaming and batch ingestion. Maybe we can extract this logic as a utility method?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-797648577


   @capistrant This looks really interesting and it has 2 approvals, so it should be good to merge. Any chance you can fix up the conflicts so we can merge this? 
   
   I'll be reading through the PR over the next week to get a better understanding of how this works, but this comment / my review shouldn't be a blocker to getting this accepted :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-816019532


   was able to play around with the latest code in my sandbox environment yesterday. It behaves how I would expect with the latest compaction changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r545293652



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.info(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            synchronized (availabilityCondition) {
+              segmentsToWaitFor.remove(s);

Review comment:
       I ended up trying a countdown latch with timeout since I am more familiar with those from use in the past as compared to your suggestion. let me know what you think of it




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-745628501


   > @capistrant thanks for the PR. This idea seems useful. I will take a look soon.
   
   Thanks @jihoonson. I'm a little worried about concurrency issues in AbstractBatchIndexTask#waitForSegmentAvailability so if your review can take extra care there that would be greatly appreciated


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r544413791



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()

Review comment:
       ya, that makes sense. Will add




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r555388018



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(

Review comment:
       My initial thoughts:
   * I agree that the code is working to achieve the same goal. so re-use would be nice
   * IndexTask appears to be straightforward if we were to take this approach.
   * Parallel Indexing and Hadoop Indexing do not seem as straightforward the way I understand things
     * parallel: we'd like to do this handoff wait from the supervisor, IMO. However there is no existing utilization of Appenderator or AppendoratorDriver at this level of parallel indexing.
     * hadoop: Hadoop does not use Appendorator or AppendoratorDriver so we would have to bolt that on with an Appendorator impl that is pretty much only there for show since hadoop indexing does not have any use other than the fact that it would be needed for this implementation (as far as I can tell)
    
   I'm sure I could make it work eventually. But I wonder if the jumping through hoops to get the task code ready to be compatible for using the registerHandoff in BaseAppenderatorDriver is worth it both in terms of effort and code legibility.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603513473



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {
+      log.warn("Asked to wait for availability for <= 0 seconds?! Requested waitTimeout: [%s]", waitTimeout);
+      return false;
+    }
+    log.info("Waiting for [%d] segments to be loaded by the cluster...", segmentsToWaitFor.size());
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+    ExecutorService exec = Execs.directExecutor();
+    CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.debug(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            doneSignal.countDown();
+          }
+      );
+    }
+
+    try {
+      return doneSignal.await(waitTimeout, TimeUnit.MILLISECONDS);
+    }
+    catch (InterruptedException e) {
+      log.warn("Interrupted while waiting for segment availablity; Unable to confirm availability!");
+      Thread.currentThread().interrupt();
+      return false;
+    }
+    finally {
+      notifier.close();

Review comment:
       I agree with this thought. Unsure on how exact implementation should look. Pasting a block below before pushing it:
   
   ```
   try(
           SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
                                                    .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource())
       ) {
   
         ExecutorService exec = Execs.directExecutor();
         CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
   
         notifier.start();
         for (DataSegment s : segmentsToWaitFor) {
           notifier.registerSegmentHandoffCallback(
               new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
               exec,
               () -> {
                 log.debug(
                     "Confirmed availability for [%s]. Removing from list of segments to wait for",
                     s.getId()
                 );
                 doneSignal.countDown();
               }
           );
         }
   
         try {
           return doneSignal.await(waitTimeout, TimeUnit.MILLISECONDS);
         }
         catch (InterruptedException e) {
           log.warn("Interrupted while waiting for segment availablity; Unable to confirm availability!");
           Thread.currentThread().interrupt();
           return false;
         }
       }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r551662522



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(

Review comment:
       Wondering if you can reuse `StreamAppenderatorDriver.registerHandoff()` or `StreamAppenderatorDriver.publishAndRegisterHandoff()` as they seem pretty similar to this new method. You would need to move that method out to `BaseAppenderatorDriver`.

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/IngestionStatsAndErrorsTaskReportData.java
##########
@@ -41,17 +41,22 @@
   @Nullable
   private String errorMsg;
 
+  @JsonProperty
+  private boolean segmentAvailabilityConfirmed;
+
   public IngestionStatsAndErrorsTaskReportData(
       @JsonProperty("ingestionState") IngestionState ingestionState,
       @JsonProperty("unparseableEvents") Map<String, Object> unparseableEvents,
       @JsonProperty("rowStats") Map<String, Object> rowStats,
-      @JsonProperty("errorMsg") @Nullable String errorMsg
+      @JsonProperty("errorMsg") @Nullable String errorMsg,
+      @JsonProperty("segmentAvailabilityConfirmed") boolean segmentAvailabilityConfirmed

Review comment:
       Can you give some more details on how this will be used in your application? Do you want to track handoff failures of each task? I'm wondering if handoff time is also important.

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
##########
@@ -1058,7 +1058,8 @@ private synchronized void persistSequences() throws IOException
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                false

Review comment:
       Why is this always `false`? Does it make more sense to be always `true` because realtime tasks will fail when handoff fails?

##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/TaskReportSerdeTest.java
##########
@@ -55,7 +55,8 @@ public void testSerde() throws Exception
             ImmutableMap.of(
                 "number", 1234
             ),
-            "an error message"
+            "an error message",
+            false

Review comment:
       Testing with `true` would be better because missing booleans are defaulted to false by Jackson in Druid.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601884079



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       I doubt it. I don't think I was writing this with the intention of having compaction tasks support this. It really wouldn't add value since compaction isn't materially changing the underlying data. But I guess since compaction is spinning up a parallel indexing task, we kinda need to consider it so I will have to check on this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r604468403



##########
File path: integration-tests/src/test/java/org/apache/druid/tests/indexer/ITCombiningFirehoseFactoryIndexTest.java
##########
@@ -59,13 +61,28 @@ public void testIndexData() throws Exception
           throw new RuntimeException(e);
         }
       };
+      final Function<String, String> transform = spec -> {
+        try {
+          return StringUtils.replace(
+              spec,
+              "%%SEGMENT_AVAIL_TIMEOUT_MILLIS%%",
+              jsonMapper.writeValueAsString("0")

Review comment:
       > Some comments on this pattern since I've seen it across a few tests
   > 
   > * Does this change mean we lose coverage on a missing `SEGMENT_AVAIL_TIMEOUT_MILLIS`? Since this is the default mode, it would be good to have coverage for that missing change
   
   There IT tests out there that indirectly test this. By indirectly, I mean that I did not add the tests but the tests do not have the config in the tuningConfig. For instance, https://github.com/capistrant/incubator-druid/blob/batch-ingest-wait-for-handoff/integration-tests/src/test/java/org/apache/druid/tests/indexer/ITIndexerTest.java#L146, is an index job that doesn't add the transform and the underlying spec file does not have the config in it. https://github.com/capistrant/incubator-druid/blob/batch-ingest-wait-for-handoff/integration-tests/src/test/resources/indexer/wikipedia_reindex_task.json#L49
   
   > * Do we have / want tests where this is set to a high enough number that we don't need a re-try loop in the integration tests while waiting for the segments to be available
   
   technically that retry loop is redundant in the case that the test uses `true,true` for the `Pair` object that determines if we should check the ingestion report for `true` as the value for the handoff. this would be an example of that, https://github.com/capistrant/incubator-druid/blob/batch-ingest-wait-for-handoff/integration-tests/src/test/java/org/apache/druid/tests/indexer/ITIndexerTest.java#L205
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601926994



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       Interesting point. I think there are some things we should think about first.
   
   - It's true that currently compaction doesn't change the underlying data much, but it can make some changes such as filtering out some unnecessary dimensions or adding new metrics. You can also change the query granularity now. In the future, I can imagine that you can even transform your data using compaction with a new support for transformSpec.
   - The compaction task is a bit special and different from other batch tasks in how it publishes segments. All other batch tasks can push segments in the middle of indexing, but should publish all those segments at the end of indexing. However, the compaction task can process each time chunk at a time when there is no change in segment granularity. In this case, it can publish segments whenever it finishes processing individual time chunk. It can also go through all time chunks even when there are some time chunks that it fails to compact. The final task status will be `FAILED` when it succeeds to compact only some time chunks but fails for others.
   - Compacting datasources is usually not the single-shot type job. Rather, you would run multiple small compaction tasks over time as in auto compaction. In that case, you would want to know what time chunks are compacted and what are not, so that you can determine what result you can get when you query certain time chunks. For the compaction that is manually set up outside druid, tracking of individual compaction tasks could be useful for this purpose. However, for auto compaction, it won't provide much value since compaction tasks are submitted by the coordinator not users. So, we need another way such as adding a new coordinator API that returns such compaction status.
   
   From these, we would probably want something similar but different for compaction from the one proposed here. I would suggest to do it in a different PR. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r551135600



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.info(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            synchronized (availabilityCondition) {
+              segmentsToWaitFor.remove(s);

Review comment:
       Looks good to me @capistrant. Thanks.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] maytasm commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
maytasm commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-784844642


   > hey @maytasm checking in to see if you have been able to look at the code at all after talking to Jihoon?
   
   Sorry I been a little bit busy. I should be able to get to this by the end of this week. Thank you for your patience. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r552223521



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/TaskReportSerdeTest.java
##########
@@ -55,7 +55,8 @@ public void testSerde() throws Exception
             ImmutableMap.of(
                 "number", 1234
             ),
-            "an error message"
+            "an error message",
+            false

Review comment:
       makes sense, will change




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603470961



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {
+      log.warn("Asked to wait for availability for <= 0 seconds?! Requested waitTimeout: [%s]", waitTimeout);
+      return false;
+    }
+    log.info("Waiting for [%d] segments to be loaded by the cluster...", segmentsToWaitFor.size());
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+    ExecutorService exec = Execs.directExecutor();
+    CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(

Review comment:
       Ah perfect! Thanks for the explanation




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601891400



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {

Review comment:
       updated for the `<` case only. Like you said, it shouldn't ever happen, but someday in the future if it does somehow this should make troubleshooting easier for an operator




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-769445603


   @capistrant I asked @maytasm to take a look.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r607435710



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java
##########
@@ -146,7 +151,7 @@
   @Nullable
   private final ClientCompactionTaskGranularitySpec granularitySpec;
   @Nullable
-  private final ParallelIndexTuningConfig tuningConfig;
+  private final CompactionTuningConfig tuningConfig;

Review comment:
       I take this back. `getTuningConfig` deals with the different types of tuningConfigs. 🤦 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603516754



##########
File path: integration-tests/src/test/java/org/apache/druid/tests/indexer/AbstractITBatchIndexTest.java
##########
@@ -280,6 +334,18 @@ private void submitTaskAndWait(
       );
     }
 
+    if (segmentAvailabilityConfirmationPair.lhs != null && segmentAvailabilityConfirmationPair.lhs) {
+      TaskReport reportRaw = indexer.getTaskReport(taskID).get("ingestionStatsAndErrors");
+      IngestionStatsAndErrorsTaskReport report = (IngestionStatsAndErrorsTaskReport) reportRaw;
+      IngestionStatsAndErrorsTaskReportData reportData = (IngestionStatsAndErrorsTaskReportData) report.getPayload();
+      if (segmentAvailabilityConfirmationPair.rhs != null) {
+        Assert.assertEquals(
+            Boolean.valueOf(reportData.isSegmentAvailabilityConfirmed()),
+            segmentAvailabilityConfirmationPair.rhs
+        );
+      }
+    }
+

Review comment:
       trying to track down exactly what retry loop you are referring to. When I click that link my browser is just loading at the top of the page so I'm not sure exactly where you are wanting me to look. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-769424545


   @jihoonson Thanks for the help and review. I will respond to your comments in that thread about future work for this.
   
   I'll work on trying to find another committer to review soon. Maybe I can trade a review for a review with someone


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603598513



##########
File path: integration-tests/src/test/java/org/apache/druid/tests/indexer/AbstractITBatchIndexTest.java
##########
@@ -280,6 +334,18 @@ private void submitTaskAndWait(
       );
     }
 
+    if (segmentAvailabilityConfirmationPair.lhs != null && segmentAvailabilityConfirmationPair.lhs) {
+      TaskReport reportRaw = indexer.getTaskReport(taskID).get("ingestionStatsAndErrors");
+      IngestionStatsAndErrorsTaskReport report = (IngestionStatsAndErrorsTaskReport) reportRaw;
+      IngestionStatsAndErrorsTaskReportData reportData = (IngestionStatsAndErrorsTaskReportData) report.getPayload();
+      if (segmentAvailabilityConfirmationPair.rhs != null) {
+        Assert.assertEquals(
+            Boolean.valueOf(reportData.isSegmentAvailabilityConfirmed()),
+            segmentAvailabilityConfirmationPair.rhs
+        );
+      }
+    }
+

Review comment:
       Maybe this link is better https://github.com/apache/druid/blob/master/integration-tests/src/test/java/org/apache/druid/tests/indexer/AbstractITBatchIndexTest.java#L308-L312
   
   ```
       if (waitForSegmentsToLoad) {
         ITRetryUtil.retryUntilTrue(
             () -> coordinator.areSegmentsLoaded(dataSourceName), "Segment Load"
         );
       }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601980096



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       @jihoonson when it comes to judging segment handoff, isn't a compaction task a parallel indexing task under the hood? I just tested a compaction task submit with tuning config and the new config for handoff set to 5 minutes. The task ran and logs indicate it confirmed handoff of newly created compacted segment. 
   
   Should not allow the handoff to occur when it is a compact type task? I'm not sure how we would do that, since the TuningConfig is not modifiable as written and if the task(s) ran by the compact task are just parallel indexing tasks, I'm not sure how we would tell them not to honor the handoff wait config




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603523465



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       I like this idea of it failing. I will work on changes as soon as I can, hopefully early this week. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r552223417



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
##########
@@ -1058,7 +1058,8 @@ private synchronized void persistSequences() throws IOException
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                false

Review comment:
       well I guess this would be true if the task succeeds and false if it does not instead of hardcoded. I was tunnel visioned on this part thinking only what my team needed and not what actual behavior should be for all different types of ingestion. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r556922336



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AppenderatorDriverRealtimeIndexTask.java
##########
@@ -584,7 +584,8 @@ protected boolean isFirehoseDrainableByClosing(FirehoseFactory firehoseFactory)
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                false

Review comment:
       I think this should be errorMsg == null. also add javadoc 
   ```
   /**
      * Return a map of reports for the task.
      *
      * A successfull task should always have a null errorMsg. A falied task should always have a non-null
      * errorMsg. Nullable error message for the task. null if task succeeded.
      *
      * @return Map of reports for the task.
      */
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r552230974



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(

Review comment:
       interesting suggestion. definitely open to it. I'll take a look as soon as I have time




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r544091714



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()

Review comment:
       nit: Each time we call the `waitForSegmentAvailability` function will communicate with coordinator no matter what the value of `waitTimeout ` is and bring extra pressure to the Coordiantor, because there may hundreds of thousands of batch tasks per day. 
   Maybe we can do a **double check** here, like when `waitTimeout<=0` then skip all the `waitForSegmentAvailability ` if possible , just in case that call `waitForSegmentAvailability` function without checking `waitTimeout`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r544412805



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.info(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            synchronized (availabilityCondition) {
+              segmentsToWaitFor.remove(s);

Review comment:
       yes, I've been trying to come up with an alternative to avoid concurrent access risks. thanks so much for this idea




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r564942219



##########
File path: server/src/main/java/org/apache/druid/segment/handoff/CoordinatorBasedSegmentHandoffNotifier.java
##########
@@ -17,7 +17,7 @@
  * under the License.
  */
 
-package org.apache.druid.segment.realtime.plumber;
+package org.apache.druid.segment.handoff;

Review comment:
       I think we should change the log level of some info logs to debug in this class which are printed per segment. It could be a lot in batch ingestion.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r565480253



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(

Review comment:
       > Good point. I agree. I think it would be better to not do such refactoring for parallel or hadoop task in this PR. But it would be still nice to reuse the same logic in both streaming and batch ingestion. Maybe we can extract this logic as a utility method?
   
   Hm, I guess I'm a little bit confused on this comment. which logic are you suggesting be shared? Since the code in StreamAppenderatorDriver is so closely coupled with the appenderator concept, I struggle to see what can be extracted. The callback function required for the appenderator is incompatible with batch ingestion as it stands today. Are you suggesting that we use the same method for both but use different callback implementations based on the ingestion type? I guess I don't understand the value there if that is the case. Otherwise I may be missing your point entirely and just need a nudge in the right direction




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601882087



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
##########
@@ -1066,7 +1075,8 @@ private synchronized void persistSequences() throws IOException
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                errorMsg == null

Review comment:
       yes, it is saying that a successful streaming task means segments were confirmed to be handed off. an unsuccessful task means we are not positive that segments were handed off. I'm unsure if it is possible to have a streaming task confirm that segments are handed off and still fail, but I assume it is not very common.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r552240383



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
##########
@@ -1058,7 +1058,8 @@ private synchronized void persistSequences() throws IOException
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                false

Review comment:
       I went with `errorMsg == null` and added a javadoc stating the errorMsg String should be null on success and non-null on failure. Current code follows that pattern




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r564932376



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
##########
@@ -1058,7 +1058,8 @@ private synchronized void persistSequences() throws IOException
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                false

Review comment:
       Oh yeah, I thought `IngestionStatsAndErrorsTaskReportData` is available only when the task succeeded. I think you are correct.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s merged pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s merged pull request #10676:
URL: https://github.com/apache/druid/pull/10676


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-802250931


   Thanks @maytasm and @jihoonson for the reviews! @suneet-s I have merged master back into my branch and pushed a small commit to fix compilation after that merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r602001680



##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       > @jihoonson when it comes to judging segment handoff, isn't a compaction task a parallel indexing task under the hood? I just tested a compaction task submit with tuning config and the new config for handoff set to 5 minutes. The task ran and logs indicate it confirmed handoff of newly created compacted segment.
   
   It is the parallel index task under the hood, but [one compaction task can run multiple parallel indexing tasks](https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java#L420-L447). With the changes in this PR, each of those parallel indexing tasks will wait for segment handoff independently. The handoff wait timeout will also apply separately for each parallel task. I think this is OK as long as it's documented, but what I meant is, maybe we could do this differently, such as waiting for segments to be handed off all together after running all parallel tasks. 
   
   > Should not allow the handoff to occur when it is a compact type task? I'm not sure how we would do that, since the TuningConfig is not modifiable as written and if the task(s) ran by the compact task are just parallel indexing tasks, I'm not sure how we would tell them not to honor the handoff wait config
   
   It seems reasonable to me to not support until we figure out what we want to do. An easy way to do so is making a copy of tuningConfig in compaction task but with `awaitSegmentAvailabilityTimeoutMillis` always being set to 0. We should document that compaction tasks don't wait for handoff if you want to do this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] suneet-s commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
suneet-s commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601822317



##########
File path: server/src/main/java/org/apache/druid/segment/indexing/TuningConfig.java
##########
@@ -40,6 +40,7 @@
   int DEFAULT_MAX_SAVED_PARSE_EXCEPTIONS = 0;
   int DEFAULT_MAX_ROWS_IN_MEMORY = 1_000_000;
   boolean DEFAULT_SKIP_BYTES_IN_MEMORY_OVERHEAD_CHECK = false;
+  int DEFAULT_AWAIT_SEGMENT_AVAILABILITY_TIMEOUT_MILLIS = 0;

Review comment:
       nit: This is a long elsewhere. I don't think this matters though.
   
   ```suggestion
     long DEFAULT_AWAIT_SEGMENT_AVAILABILITY_TIMEOUT_MILLIS = 0;
   ```

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/IngestionStatsAndErrorsTaskReportData.java
##########
@@ -41,17 +41,22 @@
   @Nullable
   private String errorMsg;
 
+  @JsonProperty
+  private boolean segmentAvailabilityConfirmed;
+
   public IngestionStatsAndErrorsTaskReportData(
       @JsonProperty("ingestionState") IngestionState ingestionState,
       @JsonProperty("unparseableEvents") Map<String, Object> unparseableEvents,
       @JsonProperty("rowStats") Map<String, Object> rowStats,
-      @JsonProperty("errorMsg") @Nullable String errorMsg
+      @JsonProperty("errorMsg") @Nullable String errorMsg,
+      @JsonProperty("segmentAvailabilityConfirmed") boolean segmentAvailabilityConfirmed

Review comment:
       Naive question: Should this be
   ```suggestion
         @JsonProperty("segmentAvailabilityConfirmed") @Nullable Boolean segmentAvailabilityConfirmed
   ```
   
   Seeing the json properties automatically make me think about version mismatches - but I don't exactly know how this is used - so I'm just asking in the hope you can save me some time from digging :)

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");

Review comment:
       nit: The question mark in this message makes it seem like this is an un-expected state. Should this be logged at a warn level instead?
   
   Honestly, if I didn't read the question mark at the end, I would have thought this is a reasonable info level message.

##########
File path: website/.spelling
##########
@@ -970,6 +970,7 @@ InputSplit
 JobHistory
 a.example.com
 assumeGrouped
+awaitSegmentAvailabilityTimeoutMillis

Review comment:
       nit: did you mean to add this twice?
   
   ```suggestion
   ```

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {

Review comment:
       This shouldn't be a warn message since the default is 0. Maybe you want a less than check, which should technically never happen, since there is a check in the constructor.
   
   ```suggestion
       } else if (waitTimeout < 0) {
   ```

##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/IndexTaskSerdeTest.java
##########
@@ -267,6 +272,7 @@ public void testBestEffortRollupWithHashedPartitionsSpec()
         true,
         10,
         100,
+        null,
         null

Review comment:
       Do we need an `IndexTaskSerdeTest` where `awaitSegmentAvailabilityTimeoutMillis` is not null

##########
File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/CompactionTaskTest.java
##########
@@ -1447,6 +1452,7 @@ private void assertIngestionSchema(
             null,
             null,
             null,
+            null,
             null

Review comment:
       Do we need a `CompactionTaskTest` where `awaitSegmentAvailabilityTimeoutMillis` is non null

##########
File path: integration-tests/src/test/java/org/apache/druid/tests/coordinator/duty/ITAutoCompactionTest.java
##########
@@ -376,6 +376,11 @@ private void loadData(String indexTask) throws Exception
   {
     String taskSpec = getResourceAsString(indexTask);
     taskSpec = StringUtils.replace(taskSpec, "%%DATASOURCE%%", fullDatasourceName);
+    taskSpec = StringUtils.replace(
+        taskSpec,
+        "%%SEGMENT_AVAIL_TIMEOUT_MILLIS%%",
+        jsonMapper.writeValueAsString("0")

Review comment:
       Is it worth testing what happens when this is non zero?
   
   Do we also want to check that the IngestionStatsAndErrorsTaskReportData reports `segmentAvailabilityConfirmed` as false now?

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {
+      log.warn("Asked to wait for availability for <= 0 seconds?! Requested waitTimeout: [%s]", waitTimeout);
+      return false;
+    }
+    log.info("Waiting for [%d] segments to be loaded by the cluster...", segmentsToWaitFor.size());
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+    ExecutorService exec = Execs.directExecutor();
+    CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(

Review comment:
       Another naive question: Is it possible for the segments to be handed off before the callback was registered?
   
   I haven't dug deep into this part of Druid yet, so I'm just if it's something you've considered

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {
+      log.warn("Asked to wait for availability for <= 0 seconds?! Requested waitTimeout: [%s]", waitTimeout);
+      return false;
+    }
+    log.info("Waiting for [%d] segments to be loaded by the cluster...", segmentsToWaitFor.size());
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+    ExecutorService exec = Execs.directExecutor();
+    CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.debug(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            doneSignal.countDown();
+          }
+      );
+    }
+
+    try {
+      return doneSignal.await(waitTimeout, TimeUnit.MILLISECONDS);
+    }
+    catch (InterruptedException e) {
+      log.warn("Interrupted while waiting for segment availablity; Unable to confirm availability!");
+      Thread.currentThread().interrupt();
+      return false;
+    }
+    finally {
+      notifier.close();

Review comment:
       Instead of catching this in the finally block, I think a try-with-resource pattern would be safer. I don't think anything can fail in the for loop where we're registering callbacks, but if it does - the SegmentHandoffNotifier won't clean up after itself
   
   ```
   try (SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory().createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource())) {
       // register handoffs and wait for signal
   }
   
   ```

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
##########
@@ -1066,7 +1075,8 @@ private synchronized void persistSequences() throws IOException
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                errorMsg == null

Review comment:
       I think we should add a comment here that says something to the effect of "For streaming ingestion, segments are considered to be available immediately if the task is successful. If the task failed, the segments are not available"
   
   At least that's what I think this is trying to say.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r603513473



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -566,6 +574,65 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {
+      log.warn("Asked to wait for availability for <= 0 seconds?! Requested waitTimeout: [%s]", waitTimeout);
+      return false;
+    }
+    log.info("Waiting for [%d] segments to be loaded by the cluster...", segmentsToWaitFor.size());
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+    ExecutorService exec = Execs.directExecutor();
+    CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.debug(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            doneSignal.countDown();
+          }
+      );
+    }
+
+    try {
+      return doneSignal.await(waitTimeout, TimeUnit.MILLISECONDS);
+    }
+    catch (InterruptedException e) {
+      log.warn("Interrupted while waiting for segment availablity; Unable to confirm availability!");
+      Thread.currentThread().interrupt();
+      return false;
+    }
+    finally {
+      notifier.close();

Review comment:
       @suneet-s  I agree with this thought. Unsure on how exact implementation should look. Pasting a block below before pushing it:
   
   ```
   try(
           SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
                                                    .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource())
       ) {
   
         ExecutorService exec = Execs.directExecutor();
         CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
   
         notifier.start();
         for (DataSegment s : segmentsToWaitFor) {
           notifier.registerSegmentHandoffCallback(
               new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
               exec,
               () -> {
                 log.debug(
                     "Confirmed availability for [%s]. Removing from list of segments to wait for",
                     s.getId()
                 );
                 doneSignal.countDown();
               }
           );
         }
   
         try {
           return doneSignal.await(waitTimeout, TimeUnit.MILLISECONDS);
         }
         catch (InterruptedException e) {
           log.warn("Interrupted while waiting for segment availablity; Unable to confirm availability!");
           Thread.currentThread().interrupt();
           return false;
         }
       }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on pull request #10676:
URL: https://github.com/apache/druid/pull/10676#issuecomment-784410961


   hey @maytasm checking in to see if you have been able to look at the code at all after talking to Jihoon? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r545293816



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java
##########
@@ -491,6 +497,31 @@ private boolean isParallelMode()
     return isParallelMode(baseInputSource, ingestionSchema.getTuningConfig());
   }
 
+  /**
+   * Attempt to wait for indexed segments to become available on the cluster.
+   * @param reportsMap Map containing information with published segments that we are going to wait for.
+   */
+  private void waitForSegmentAvailability(Map<String, PushedSegmentsReport> reportsMap)
+  {
+    ArrayList<DataSegment> segmentsToWaitFor = new ArrayList<>();
+    reportsMap.values()
+              .forEach(report -> {
+                segmentsToWaitFor.addAll(report.getNewSegments());
+              });
+    ExecutorService availabilityExec = Execs.singleThreaded("ParallelTaskAvailabilityWaitExec");
+    try {
+      segmentAvailabilityConfirmationCompleted = waitForSegmentAvailability(
+          toolbox,
+          availabilityExec,
+          segmentsToWaitFor,
+          ingestionSchema.getTuningConfig().getAwaitSegmentAvailabilityTimeoutMillis()

Review comment:
       I'm fine with that. addressed in latest commits




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r566368729



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -580,6 +588,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return true;
+    } else if (waitTimeout <= 0) {
+      log.warn("Asked to wait for availability for <= 0 seconds?! Requested waitTimeout: [%s]", waitTimeout);
+      return false;
+    }
+    log.info("Waiting for [%d] segments to be loaded by the cluster...", segmentsToWaitFor.size());
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()
+                                             .createSegmentHandoffNotifier(segmentsToWaitFor.get(0).getDataSource());
+    ExecutorService exec = Execs.directExecutor();
+    CountDownLatch doneSignal = new CountDownLatch(segmentsToWaitFor.size());
+
+    notifier.start();
+    for (DataSegment s : segmentsToWaitFor) {
+      notifier.registerSegmentHandoffCallback(
+          new SegmentDescriptor(s.getInterval(), s.getVersion(), s.getShardSpec().getPartitionNum()),
+          exec,
+          () -> {
+            log.debug(
+                "Confirmed availability for [%s]. Removing from list of segments to wait for",
+                s.getId()
+            );
+            doneSignal.countDown();
+          }
+      );
+    }
+
+    try {
+      return doneSignal.await(waitTimeout, TimeUnit.MILLISECONDS);
+    }
+    catch (InterruptedException e) {

Review comment:
       The interrupted state is cleared out after it's checked. We should set the state of the current thread back. Please add `Thread.currentThread().interrupt()`.

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(

Review comment:
       Yeah, I think you are right.

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/IngestionStatsAndErrorsTaskReportData.java
##########
@@ -41,17 +41,22 @@
   @Nullable
   private String errorMsg;
 
+  @JsonProperty
+  private boolean segmentAvailabilityConfirmed;
+
   public IngestionStatsAndErrorsTaskReportData(
       @JsonProperty("ingestionState") IngestionState ingestionState,
       @JsonProperty("unparseableEvents") Map<String, Object> unparseableEvents,
       @JsonProperty("rowStats") Map<String, Object> rowStats,
-      @JsonProperty("errorMsg") @Nullable String errorMsg
+      @JsonProperty("errorMsg") @Nullable String errorMsg,
+      @JsonProperty("segmentAvailabilityConfirmed") boolean segmentAvailabilityConfirmed

Review comment:
       > For us a simple yes/no will suffice. The cluster operators would have the goal of having 100% of jobs successfully handoff data before the timeout, but when that doesn't happen our users simply want to know that they may need to wait longer. We are simply trying to be transparent and report the point in time status. The onus of finding out when the data is fully loaded if this timeout expired before loading, would fall on a different solution (TBD).
   
   Cool, are you working on "the different solution"? That would be interesting too.
   
   > Why the handoff failed would be something I as an operator am more interested compared to a user (unless that user is also an operator). I think that would be very difficult to communicate in these reports since the indexing task doesn't know much about what the rest of the cluster is doing.
   
   I agree. I think we need more visibility on the coordinator behavior.
   
   > Knowing how long it took before the time out could be found in the spec, but I guess it could be useful to add that value to the report as well if you think users would want to have quick reference. I think that rather than having that static value, it could be cool to have the dynamic time waited for handoff. Maybe it is the static value because we hit the timeout. but as an operator I would enjoy seeing how long each successful job waited for handoff. what do you think about that?
   
   That seems useful to me too :+1: 
   
   For the time to fail handoff, due to the above issue of the lack of ability to know the cause of handoff failures, I guess I was wondering if the report can be a false alarm. For example, the report can say it failed to confirm the segments handed off, but maybe the handoff could be even not triggered at all for some reason. I don't think this can happen for now, but is possible in the future if someone else modifies this area for some good reason. `segmentAvailabilityConfirmationCompleted` + time to fail handoff can be an indicator of such unexpected failures. I would say this is not a blocker for this PR, but it seems useful to me.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r601875629



##########
File path: website/.spelling
##########
@@ -970,6 +970,7 @@ InputSplit
 JobHistory
 a.example.com
 assumeGrouped
+awaitSegmentAvailabilityTimeoutMillis

Review comment:
       The way I understood it is that each file has it's own dictionary. The first is in the hadoop index file section and the second is in the native index file section




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] zhangyue19921010 commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r544091714



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +582,73 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(TaskToolbox toolbox, ExecutorService exec, List<DataSegment> segmentsToWaitFor, long waitTimeout)
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.info("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;
+    }
+    log.info("Waiting for segments to be loaded by the cluster...");
+
+    SegmentHandoffNotifier notifier = toolbox.getSegmentHandoffNotifierFactory()

Review comment:
       nit: Each time we call the `waitForSegmentAvailability` function will communicate with coordinator no matter what the value of `waitTimeout` is and bring extra pressure to the Coordiantor, because there may be hundreds of thousands of batch tasks per day. 
   Maybe we can do a **double check** here, like when `waitTimeout<=0` then skip all the `waitForSegmentAvailability ` if possible , just in case that call `waitForSegmentAvailability` function without checking `waitTimeout`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] abhishekagarwal87 commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r551136043



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity findGranularityFromSegments(List<DataSegment> segments)
     }
   }
 
+  /**
+   * Wait for segments to become available on the cluster. If waitTimeout is reached, giveup on waiting. This is a
+   * QoS method that can be used to make Batch Ingest tasks wait to finish until their ingested data is available on
+   * the cluster. Doing so gives an end user assurance that a Successful task status means their data is available
+   * for querying.
+   *
+   * @param toolbox {@link TaskToolbox} object with for assisting with task work.
+   * @param segmentsToWaitFor {@link List} of segments to wait for availability.
+   * @param waitTimeout Millis to wait before giving up
+   * @return True if all segments became available, otherwise False.
+   */
+  protected boolean waitForSegmentAvailability(
+      TaskToolbox toolbox,
+      List<DataSegment> segmentsToWaitFor,
+      long waitTimeout
+  )
+  {
+    if (segmentsToWaitFor.isEmpty()) {
+      log.warn("Asked to wait for segments to be available, but I wasn't provided with any segments!?");
+      return false;

Review comment:
       shouldn't this return true here? If there are no segments to confirm, the confirmation is done. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

Posted by GitBox <gi...@apache.org>.
capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r556922336



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AppenderatorDriverRealtimeIndexTask.java
##########
@@ -584,7 +584,8 @@ protected boolean isFirehoseDrainableByClosing(FirehoseFactory firehoseFactory)
                 ingestionState,
                 getTaskCompletionUnparseableEvents(),
                 getTaskCompletionRowStats(),
-                errorMsg
+                errorMsg,
+                false

Review comment:
       I think this should be errorMsg == null




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org