[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #8454: Web console: Fix segment re-ingest

[GitBox <gi...@apache.org>]

vogievetsky commented on a change in pull request #8454: Web console: Fix segment re-ingest
URL: https://github.com/apache/incubator-druid/pull/8454#discussion_r320354166
 
 

 ##########
 File path: web-console/src/utils/sampler.ts
 ##########
 @@ -167,17 +178,66 @@ function makeSamplerIoConfig(
   return ioConfig;
 }
 
+/**
+ * This function scopes down the interval of an ingestSegment firehose for the data sampler
+ * this is needed because the ingestSegment firehose gets the interval you are sampling over,
+ * looks up the corresponding segments and segment locations from metadata store, downloads
+ * every segment from deep storage to disk, and then maps all the segments into memory;
+ * and this happens in the constructor before the timer thread is even created meaning the sampler
+ * will time out on a larger interval.
+ * This is essentially a workaround for https://github.com/apache/incubator-druid/issues/8448
+ * @param ioConfig The IO Config to scope down the interval of
+ */
+export async function scopeDownIngestSegmentFirehoseIntervalIfNeeded(
+  ioConfig: IoConfig,
+): Promise<IoConfig> {
+  if (deepGet(ioConfig, 'firehose.type') !== 'ingestSegment') return ioConfig;
+  const interval = deepGet(ioConfig, 'firehose.interval');
+  const intervalParts = interval.split('/');
+  const start = new Date(intervalParts[0]);
+  if (isNaN(start.valueOf())) throw new Error(`could not decode interval start`);
+  const end = new Date(intervalParts[1]);
+  if (isNaN(end.valueOf())) throw new Error(`could not decode interval end`);
+
+  // Less than or equal to 1 hour so there is no need to adjust intervals
 
 Review comment:
   The comment was a typo. I started with making the interval 1 hr then changed it to 1/2 hour. That number came from the fact that when I did 1hr because my adjuster was based on maxTime it would not be hour aligned thus it would capture the segments from 2 hours, setting the interval to 1/2 seems to work better as it reduces the probability that it would capture segments from 2 hours. I just tested it empirically on the largest datasources accessible to me.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org