You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Fu Xiang <fx...@gmail.com> on 2020/02/15 03:55:00 UTC
Some thoughts on SegmentPreProcessor
1. For large table, the segment reloading may take very long time during
server start up. Shall we consider load segment as it is during server
startup, then reload each segment afterward?
2 .Shall we consider when to regenerate the segment. E,g. indexing removal,
update dictionary type from dict to non-dict (not yet supported).
--
Xiang Fu
Re: Some thoughts on SegmentPreProcessor
Posted by Fu Xiang <fx...@gmail.com>.
For 1, It’s more about getting server startup faster. Otherwise it may take
long time to do rolling restart.
For 2, we can keep adding per column basis index to existing segment, just
need to support dictionary and non-dictionary conversion on the fly.
Meanwhile we can keep track of real size for used index size as well as
segment total size to decide when to do compaction.
On Sat, Feb 15, 2020 at 11:35 Mayank Shrivastava <ma...@gmail.com>
wrote:
> Good thought, but need more clarity. For cases that warrant reload
> anyways, how does faster server startup help?
> For 2, I see this as a precursor to adaptive indexing on the fly.
>
> Sent from my iPhone
>
> > On Feb 14, 2020, at 7:55 PM, Fu Xiang <fx...@gmail.com> wrote:
> >
> > 1. For large table, the segment reloading may take very long time during
> > server start up. Shall we consider load segment as it is during server
> > startup, then reload each segment afterward?
> > 2 .Shall we consider when to regenerate the segment. E,g. indexing
> removal,
> > update dictionary type from dict to non-dict (not yet supported).
> >
> > --
> > Xiang Fu
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
> For additional commands, e-mail: dev-help@pinot.apache.org
>
> --
Xiang Fu
Re: Some thoughts on SegmentPreProcessor
Posted by Mayank Shrivastava <ma...@gmail.com>.
Good thought, but need more clarity. For cases that warrant reload anyways, how does faster server startup help?
For 2, I see this as a precursor to adaptive indexing on the fly.
Sent from my iPhone
> On Feb 14, 2020, at 7:55 PM, Fu Xiang <fx...@gmail.com> wrote:
>
> 1. For large table, the segment reloading may take very long time during
> server start up. Shall we consider load segment as it is during server
> startup, then reload each segment afterward?
> 2 .Shall we consider when to regenerate the segment. E,g. indexing removal,
> update dictionary type from dict to non-dict (not yet supported).
>
> --
> Xiang Fu
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
For additional commands, e-mail: dev-help@pinot.apache.org