How a new index is built in runtime?

[Denis Magda <dm...@apache.org>]

Alex P., Vladimir,

Having CREATE INDEX command we can define indexes in runtime. However, it’s unclear how a new index is built.

Let’s imagine I have a field “name” that was in Person’s model for a while and there are millions of such objects in the cluster. Now I turned the field into the index in runtime. How Ignite is going to built the index? Do we iterate over the millions of objects with the field in background or at the time of the CREATE INDEX execution blocking the latter? Or is there more sophisticated process?

—
Denis 

Re: How a new index is built in runtime?

[Sergey Kozlov <sk...@gridgain.com>]

Vova

Is the data available during an index creation?

On Mon, Aug 28, 2017 at 3:46 PM, <ds...@apache.org> wrote:

> Vova, how hard is it to make it multi-threaded?
>
> ⁣D.​
>
>
> On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov <
> vozerov@gridgain.com> wrote:
> >Denis,
> >
> >We iterate over the whole cache and build the index entry-by-entry.
> >Control
> >is returned back to the user when index is ready.
> >
> >On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <yz...@apache.org>
> >wrote:
> >
> >> Of course, iteration should have an option to be run from more than 1
> >> thread. What will really help, IMO, is ability to insert presorted
> >batches
> >> in a single tree operation.
> >>
> >> --
> >> Yakov Zhdanov
> >>
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

Re: How a new index is built in runtime?

[Vladimir Ozerov <vo...@gridgain.com>]

Should not be too hard. However, this will not change situation a lot.
Instead, we need to implement more efficient algorithm of index filling.
E.g. fill it with pre-sorted batches. We already have a ticket for that.

On Mon, Aug 28, 2017 at 3:46 PM, <ds...@apache.org> wrote:

> Vova, how hard is it to make it multi-threaded?
>
> ⁣D.​
>
>
> On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov <
> vozerov@gridgain.com> wrote:
> >Denis,
> >
> >We iterate over the whole cache and build the index entry-by-entry.
> >Control
> >is returned back to the user when index is ready.
> >
> >On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <yz...@apache.org>
> >wrote:
> >
> >> Of course, iteration should have an option to be run from more than 1
> >> thread. What will really help, IMO, is ability to insert presorted
> >batches
> >> in a single tree operation.
> >>
> >> --
> >> Yakov Zhdanov
> >>
>

Re: How a new index is built in runtime?

[ds...@apache.org]

Vova, how hard is it to make it multi-threaded?

⁣D.​


On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov <vo...@gridgain.com> wrote:
>Denis,
>
>We iterate over the whole cache and build the index entry-by-entry.
>Control
>is returned back to the user when index is ready.
>
>On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <yz...@apache.org>
>wrote:
>
>> Of course, iteration should have an option to be run from more than 1
>> thread. What will really help, IMO, is ability to insert presorted
>batches
>> in a single tree operation.
>>
>> --
>> Yakov Zhdanov
>>

Re: How a new index is built in runtime?

[Vladimir Ozerov <vo...@gridgain.com>]

Sync.

пн, 28 авг. 2017 г. в 19:34, Denis Magda <dm...@apache.org>:

> Vladimir,
>
> Is this a sync or async operation?
>
> —
> Denis
>
> > On Aug 28, 2017, at 1:05 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
> >
> > Denis,
> >
> > We iterate over the whole cache and build the index entry-by-entry.
> Control
> > is returned back to the user when index is ready.
> >
> > On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <yz...@apache.org>
> wrote:
> >
> >> Of course, iteration should have an option to be run from more than 1
> >> thread. What will really help, IMO, is ability to insert presorted
> batches
> >> in a single tree operation.
> >>
> >> --
> >> Yakov Zhdanov
> >>
>
>

Re: How a new index is built in runtime?

[Denis Magda <dm...@apache.org>]

Vladimir,

Is this a sync or async operation?

—
Denis

> On Aug 28, 2017, at 1:05 AM, Vladimir Ozerov <vo...@gridgain.com> wrote:
> 
> Denis,
> 
> We iterate over the whole cache and build the index entry-by-entry. Control
> is returned back to the user when index is ready.
> 
> On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <yz...@apache.org> wrote:
> 
>> Of course, iteration should have an option to be run from more than 1
>> thread. What will really help, IMO, is ability to insert presorted batches
>> in a single tree operation.
>> 
>> --
>> Yakov Zhdanov
>> 

Re: How a new index is built in runtime?

[Vladimir Ozerov <vo...@gridgain.com>]

Denis,

We iterate over the whole cache and build the index entry-by-entry. Control
is returned back to the user when index is ready.

On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov <yz...@apache.org> wrote:

> Of course, iteration should have an option to be run from more than 1
> thread. What will really help, IMO, is ability to insert presorted batches
> in a single tree operation.
>
> --
> Yakov Zhdanov
>

Re: How a new index is built in runtime?

[Yakov Zhdanov <yz...@apache.org>]

Of course, iteration should have an option to be run from more than 1
thread. What will really help, IMO, is ability to insert presorted batches
in a single tree operation.

--
Yakov Zhdanov

Re: How a new index is built in runtime?

[Dmitriy Setrakyan <ds...@apache.org>]

On Thu, Aug 17, 2017 at 1:47 PM, Yakov Zhdanov <yz...@apache.org> wrote:

> Denis, you are absolutely right. Iterating through millions of objects is
> indeed a very complex task. Ignite casts a secret spell instead.\
>

Well, Ignite should cast a secret spell and do it in parallel, across
multiple threads. This will make index creation faster, but will occupy
more cores. Perhaps we should have both, single-threaded and multi-threaded
options.


>
> --Yakov
>
> 2017-08-17 21:26 GMT+01:00 Denis Magda <dm...@apache.org>:
>
> > Alex P., Vladimir,
> >
> > Having CREATE INDEX command we can define indexes in runtime. However,
> > it’s unclear how a new index is built.
> >
> > Let’s imagine I have a field “name” that was in Person’s model for a
> while
> > and there are millions of such objects in the cluster. Now I turned the
> > field into the index in runtime. How Ignite is going to built the index?
> Do
> > we iterate over the millions of objects with the field in background or
> at
> > the time of the CREATE INDEX execution blocking the latter? Or is there
> > more sophisticated process?
> >
> > —
> > Denis
>

Re: How a new index is built in runtime?

[Yakov Zhdanov <yz...@apache.org>]

Denis, you are absolutely right. Iterating through millions of objects is
indeed a very complex task. Ignite casts a secret spell instead.

--Yakov

2017-08-17 21:26 GMT+01:00 Denis Magda <dm...@apache.org>:

> Alex P., Vladimir,
>
> Having CREATE INDEX command we can define indexes in runtime. However,
> it’s unclear how a new index is built.
>
> Let’s imagine I have a field “name” that was in Person’s model for a while
> and there are millions of such objects in the cluster. Now I turned the
> field into the index in runtime. How Ignite is going to built the index? Do
> we iterate over the millions of objects with the field in background or at
> the time of the CREATE INDEX execution blocking the latter? Or is there
> more sophisticated process?
>
> —
> Denis