You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@druid.apache.org by Liran Funaro <li...@verizonmedia.com.INVALID> on 2020/08/31 12:22:22 UTC

Design Review for Configurable Index Type

I will appreciate the community feedback/review on a design proposal I
submitted: https://github.com/apache/druid/issues/10321.

This design proposal introduces an API that allows users to choose which
in-memory indexing approach to use during ingestion (e.g.,
OnheapIncrementalIndex).

As suggested by Jonathan Wei during one discussion (
https://github.com/apache/druid/issues/5698#issuecomment-506881097), this
API will allow extensions to offer their own in-memory index
implementations.

It is motivated by the understanding that different indexing approaches
might have different tradeoffs. Thus, it can be beneficial to allow users
to select the best approach for their workload. A notable example is #10001
<https://github.com/apache/druid/pull/10001> which shows significant
improvements in throughput and resource utilization for batch ingestion.

This proposal is accompanied by a work-in-progress PR:
https://github.com/apache/druid/pull/10335

Thanks,

Liran

Re: Design Review for Configurable Index Type

Posted by Jihoon Son <ji...@apache.org>.

Hi Liran,

thanks for an interesting proposal! I believe this proposal will help to
compare different implementations.
I haven't looked through the proposal closely yet, but will try soon.

On Mon, Aug 31, 2020 at 5:22 AM Liran Funaro
<li...@verizonmedia.com.invalid> wrote:

> I will appreciate the community feedback/review on a design proposal I
> submitted: https://github.com/apache/druid/issues/10321.
>
> This design proposal introduces an API that allows users to choose which
> in-memory indexing approach to use during ingestion (e.g.,
> OnheapIncrementalIndex).
>
> As suggested by Jonathan Wei during one discussion (
> https://github.com/apache/druid/issues/5698#issuecomment-506881097), this
> API will allow extensions to offer their own in-memory index
> implementations.
>
> It is motivated by the understanding that different indexing approaches
> might have different tradeoffs. Thus, it can be beneficial to allow users
> to select the best approach for their workload. A notable example is #10001
> <https://github.com/apache/druid/pull/10001> which shows significant
> improvements in throughput and resource utilization for batch ingestion.
>
> This proposal is accompanied by a work-in-progress PR:
> https://github.com/apache/druid/pull/10335
>
> Thanks,
>
> Liran
>