You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Vladimir Ozerov <vo...@gridgain.com> on 2017/08/01 10:26:59 UTC
Geo-spatial and full-text indexes support in Ignite 2.x
Guys,
AFAIK these two index types are not supported with enabled persistence at
the moment, neither they stored on the disk anyhow. Can someone help with
estimates on how difficult would it be to implement these indexes over
page-memory architecture?
Looks like we will have to write our own implementation of these indexes,
instead of relying on Lucene and H2. Am I right?
Vladimir.
Re: Geo-spatial and full-text indexes support in Ignite 2.x
Posted by Andrey Mashenkov <an...@gmail.com>.
Vladimir,
Is my understanding right you suggest to create a custom Lucene codec which
will be able to store index in ignite page memory?
Lucene javadoc has complete description of codec API [1] and looks like it
is possible to implement custom one.
[1]
https://lucene.apache.org/core/5_5_2/core/org/apache/lucene/codecs/lucene54/package-summary.html#package_description
On Wed, Aug 2, 2017 at 12:02 AM, Denis Magda <dm...@apache.org> wrote:
> Andrey,
>
> What I’m trying to say is that presently both text and geo-spatial indexes
> are unsupported NOT only for the persistence layer but for the durable
> (page) memory in general - as you properly pointed out the text indexes are
> stored in the old off-heap memory but has to be moved to be handled in page
> memory’s off-heap region directly.
>
> Makes sense?
>
> —
> Denis
>
> > On Aug 1, 2017, at 1:05 PM, Andrey Mashenkov <an...@gmail.com>
> wrote:
> >
> > Denis,
> >
> > I see we still use GridUnsafeMemory in LuceneIndex [1] in master.
> > Do you mean a wrapper to new memory manager is used here?
> >
> > [1]
> > https://github.com/apache/ignite/blob/master/modules/
> indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/
> GridLuceneIndex.java
> >
> > On Tue, Aug 1, 2017 at 10:28 PM, Denis Magda <dm...@apache.org> wrote:
> >
> >> Andrey,
> >>
> >> I’ve already wiped the old off-heap memory out of my mind. Sure I
> assumed
> >> the off-heap space managed by the new memory architecture.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov <
> >> andrey.mashenkov@gmail.com> wrote:
> >>>
> >>> Denis,
> >>>
> >>> Lucene fullText index stores off-heap in old way, not in page memory.
> >>> Therefore it is not persistent.
> >>>
> >>> There is a second issue related to FullText index as it is different
> kind
> >>> of index and it has own query type and it can't be use in SQL queries.
> >>> Looks like it make sense to integrate FullText indices in our SQL layer
> >> as
> >>> well.
> >>>
> >>> AFAIK, H2 has some support of FullText indices based on Lucene, so we
> can
> >>> get a hint how we can integrate it.
> >>>
> >>>
> >>>
> >>> On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <dm...@apache.org> wrote:
> >>>
> >>>> Vladimir,
> >>>>
> >>>> We need to consider that these two types of indexes are not stored
> >>>> off-heap either.
> >>>>
> >>>> It expands the task a bit — the indexes have to be fully integrated
> with
> >>>> the new durable memory architecture supporting both off-heap and
> >>>> persistence layers.
> >>>>
> >>>> —
> >>>> Denis
> >>>>
> >>>>> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <vo...@gridgain.com>
> >>>> wrote:
> >>>>>
> >>>>> Guys,
> >>>>>
> >>>>> AFAIK these two index types are not supported with enabled
> persistence
> >> at
> >>>>> the moment, neither they stored on the disk anyhow. Can someone help
> >> with
> >>>>> estimates on how difficult would it be to implement these indexes
> over
> >>>>> page-memory architecture?
> >>>>>
> >>>>> Looks like we will have to write our own implementation of these
> >> indexes,
> >>>>> instead of relying on Lucene and H2. Am I right?
> >>>>>
> >>>>> Vladimir.
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Best regards,
> >>> Andrey V. Mashenkov
> >>
> >>
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
>
>
--
Best regards,
Andrey V. Mashenkov
Re: Geo-spatial and full-text indexes support in Ignite 2.x
Posted by Denis Magda <dm...@apache.org>.
Andrey,
What I’m trying to say is that presently both text and geo-spatial indexes are unsupported NOT only for the persistence layer but for the durable (page) memory in general - as you properly pointed out the text indexes are stored in the old off-heap memory but has to be moved to be handled in page memory’s off-heap region directly.
Makes sense?
—
Denis
> On Aug 1, 2017, at 1:05 PM, Andrey Mashenkov <an...@gmail.com> wrote:
>
> Denis,
>
> I see we still use GridUnsafeMemory in LuceneIndex [1] in master.
> Do you mean a wrapper to new memory manager is used here?
>
> [1]
> https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridLuceneIndex.java
>
> On Tue, Aug 1, 2017 at 10:28 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Andrey,
>>
>> I’ve already wiped the old off-heap memory out of my mind. Sure I assumed
>> the off-heap space managed by the new memory architecture.
>>
>> —
>> Denis
>>
>>> On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov <
>> andrey.mashenkov@gmail.com> wrote:
>>>
>>> Denis,
>>>
>>> Lucene fullText index stores off-heap in old way, not in page memory.
>>> Therefore it is not persistent.
>>>
>>> There is a second issue related to FullText index as it is different kind
>>> of index and it has own query type and it can't be use in SQL queries.
>>> Looks like it make sense to integrate FullText indices in our SQL layer
>> as
>>> well.
>>>
>>> AFAIK, H2 has some support of FullText indices based on Lucene, so we can
>>> get a hint how we can integrate it.
>>>
>>>
>>>
>>> On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <dm...@apache.org> wrote:
>>>
>>>> Vladimir,
>>>>
>>>> We need to consider that these two types of indexes are not stored
>>>> off-heap either.
>>>>
>>>> It expands the task a bit — the indexes have to be fully integrated with
>>>> the new durable memory architecture supporting both off-heap and
>>>> persistence layers.
>>>>
>>>> —
>>>> Denis
>>>>
>>>>> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <vo...@gridgain.com>
>>>> wrote:
>>>>>
>>>>> Guys,
>>>>>
>>>>> AFAIK these two index types are not supported with enabled persistence
>> at
>>>>> the moment, neither they stored on the disk anyhow. Can someone help
>> with
>>>>> estimates on how difficult would it be to implement these indexes over
>>>>> page-memory architecture?
>>>>>
>>>>> Looks like we will have to write our own implementation of these
>> indexes,
>>>>> instead of relying on Lucene and H2. Am I right?
>>>>>
>>>>> Vladimir.
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>
>>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
Re: Geo-spatial and full-text indexes support in Ignite 2.x
Posted by Andrey Mashenkov <an...@gmail.com>.
Denis,
I see we still use GridUnsafeMemory in LuceneIndex [1] in master.
Do you mean a wrapper to new memory manager is used here?
[1]
https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridLuceneIndex.java
On Tue, Aug 1, 2017 at 10:28 PM, Denis Magda <dm...@apache.org> wrote:
> Andrey,
>
> I’ve already wiped the old off-heap memory out of my mind. Sure I assumed
> the off-heap space managed by the new memory architecture.
>
> —
> Denis
>
> > On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov <
> andrey.mashenkov@gmail.com> wrote:
> >
> > Denis,
> >
> > Lucene fullText index stores off-heap in old way, not in page memory.
> > Therefore it is not persistent.
> >
> > There is a second issue related to FullText index as it is different kind
> > of index and it has own query type and it can't be use in SQL queries.
> > Looks like it make sense to integrate FullText indices in our SQL layer
> as
> > well.
> >
> > AFAIK, H2 has some support of FullText indices based on Lucene, so we can
> > get a hint how we can integrate it.
> >
> >
> >
> > On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <dm...@apache.org> wrote:
> >
> >> Vladimir,
> >>
> >> We need to consider that these two types of indexes are not stored
> >> off-heap either.
> >>
> >> It expands the task a bit — the indexes have to be fully integrated with
> >> the new durable memory architecture supporting both off-heap and
> >> persistence layers.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <vo...@gridgain.com>
> >> wrote:
> >>>
> >>> Guys,
> >>>
> >>> AFAIK these two index types are not supported with enabled persistence
> at
> >>> the moment, neither they stored on the disk anyhow. Can someone help
> with
> >>> estimates on how difficult would it be to implement these indexes over
> >>> page-memory architecture?
> >>>
> >>> Looks like we will have to write our own implementation of these
> indexes,
> >>> instead of relying on Lucene and H2. Am I right?
> >>>
> >>> Vladimir.
> >>
> >>
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
>
>
--
Best regards,
Andrey V. Mashenkov
Re: Geo-spatial and full-text indexes support in Ignite 2.x
Posted by Denis Magda <dm...@apache.org>.
Andrey,
I’ve already wiped the old off-heap memory out of my mind. Sure I assumed the off-heap space managed by the new memory architecture.
—
Denis
> On Aug 1, 2017, at 11:14 AM, Andrey Mashenkov <an...@gmail.com> wrote:
>
> Denis,
>
> Lucene fullText index stores off-heap in old way, not in page memory.
> Therefore it is not persistent.
>
> There is a second issue related to FullText index as it is different kind
> of index and it has own query type and it can't be use in SQL queries.
> Looks like it make sense to integrate FullText indices in our SQL layer as
> well.
>
> AFAIK, H2 has some support of FullText indices based on Lucene, so we can
> get a hint how we can integrate it.
>
>
>
> On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <dm...@apache.org> wrote:
>
>> Vladimir,
>>
>> We need to consider that these two types of indexes are not stored
>> off-heap either.
>>
>> It expands the task a bit — the indexes have to be fully integrated with
>> the new durable memory architecture supporting both off-heap and
>> persistence layers.
>>
>> —
>> Denis
>>
>>> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <vo...@gridgain.com>
>> wrote:
>>>
>>> Guys,
>>>
>>> AFAIK these two index types are not supported with enabled persistence at
>>> the moment, neither they stored on the disk anyhow. Can someone help with
>>> estimates on how difficult would it be to implement these indexes over
>>> page-memory architecture?
>>>
>>> Looks like we will have to write our own implementation of these indexes,
>>> instead of relying on Lucene and H2. Am I right?
>>>
>>> Vladimir.
>>
>>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
Re: Geo-spatial and full-text indexes support in Ignite 2.x
Posted by Andrey Mashenkov <an...@gmail.com>.
Denis,
Lucene fullText index stores off-heap in old way, not in page memory.
Therefore it is not persistent.
There is a second issue related to FullText index as it is different kind
of index and it has own query type and it can't be use in SQL queries.
Looks like it make sense to integrate FullText indices in our SQL layer as
well.
AFAIK, H2 has some support of FullText indices based on Lucene, so we can
get a hint how we can integrate it.
On Tue, Aug 1, 2017 at 8:09 PM, Denis Magda <dm...@apache.org> wrote:
> Vladimir,
>
> We need to consider that these two types of indexes are not stored
> off-heap either.
>
> It expands the task a bit — the indexes have to be fully integrated with
> the new durable memory architecture supporting both off-heap and
> persistence layers.
>
> —
> Denis
>
> > On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
> >
> > Guys,
> >
> > AFAIK these two index types are not supported with enabled persistence at
> > the moment, neither they stored on the disk anyhow. Can someone help with
> > estimates on how difficult would it be to implement these indexes over
> > page-memory architecture?
> >
> > Looks like we will have to write our own implementation of these indexes,
> > instead of relying on Lucene and H2. Am I right?
> >
> > Vladimir.
>
>
--
Best regards,
Andrey V. Mashenkov
Re: Geo-spatial and full-text indexes support in Ignite 2.x
Posted by Denis Magda <dm...@apache.org>.
Vladimir,
We need to consider that these two types of indexes are not stored off-heap either.
It expands the task a bit — the indexes have to be fully integrated with the new durable memory architecture supporting both off-heap and persistence layers.
—
Denis
> On Aug 1, 2017, at 3:26 AM, Vladimir Ozerov <vo...@gridgain.com> wrote:
>
> Guys,
>
> AFAIK these two index types are not supported with enabled persistence at
> the moment, neither they stored on the disk anyhow. Can someone help with
> estimates on how difficult would it be to implement these indexes over
> page-memory architecture?
>
> Looks like we will have to write our own implementation of these indexes,
> instead of relying on Lucene and H2. Am I right?
>
> Vladimir.