What to set KafkaIndexingService taskDuration to?

[Dylan Wylie <dy...@gmail.com>]

Hey all,

With the Kafka Indexing Service now supporting incremental handoffs how are
folks configuring the taskDuration of the kafka indexing service?

If the service was ingesting hourly data into hourly segments, would there
be any downside to setting the taskDuration to one or more days? Or even
just have the task run forever?

The theorised advantage being when the task runs forever the service
doesn't abruptly write smaller segments.

Best regards,
Dylan

Re: What to set KafkaIndexingService taskDuration to?

[Gian Merlino <gi...@apache.org>]

Hi Dylan,

"Forever" is still not ideal due to the fact that task logs are not rotated
until tasks exit, and they can get unwieldy. But you can safely run with
longer durations. I haven't seen anyone doing days, but I have seen a few
people doing 4-8 hours and that works fine.

On Wed, Apr 4, 2018 at 4:25 AM, Dylan Wylie <dy...@gmail.com> wrote:

> Hey all,
>
> With the Kafka Indexing Service now supporting incremental handoffs how
> are folks configuring the taskDuration of the kafka indexing service?
>
> If the service was ingesting hourly data into hourly segments, would there
> be any downside to setting the taskDuration to one or more days? Or even
> just have the task run forever?
>
> The theorised advantage being when the task runs forever the service
> doesn't abruptly write smaller segments.
>
> Best regards,
> Dylan
>
>