You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by steve mathew <st...@gmail.com> on 2016/02/24 07:37:16 UTC
geode for specific usecase
Hello I am new to the apache geode and exploring a possibility to store
data sorted into regions.
I need to store data sorted for the scan(startkey, endkey) operation.
Simple looking at the APIs it looks geode is not designed for this. Is
there any way to achieve this?
Thanks in advance.
Kindly advise.
-Steve.
Re: geode for specific usecase
Posted by Jason Huynh <jh...@pivotal.io>.
Here is my attempt at answering overhead info about indexes in the past.
This does not include any temporary objects/garbage created when executing
a query or created when initially creating the index.
Compact Functional Index:
1.) Reference for every entry in the region
2.) Size of extracted index key
3.) Internal data structures (ConcurrentHashSet) used per index key
Functional Index
1.) Size of copy of value and extracted index key per entry to form the
tuple
2.) Size of extracted key
3.) Internal data structures per index key, tuple structure
Hash Index
1.) Reference per entry
2.) Internal array size
Map Index
1.) Size of extracted key
2.) Size of internal map
3.) Sizes of Functional or Compact Index
On Thu, Feb 25, 2016 at 9:23 AM, Michael Stolz <ms...@pivotal.io> wrote:
> If you are storing the data using pdx auto seriaization the index will work
> just fine with your custom object keys.
>
>
> There is no short answer to how much extra memory overhead an index will
> cause because it is dependent on the actual data itself (really the
> cardinality of the data). See
> https://en.wikipedia.org/wiki/Cardinality_(SQL_statements)
>
> They do tend to be large. Use indices judiciously.
>
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: 631-835-4771
>
> On Wed, Feb 24, 2016 at 12:34 PM, steve mathew <st...@gmail.com>
> wrote:
>
> > Thank you Dan.
> >
> > Will this work with custom object keys?
> > How much will be extra memory overhead when creating index?
> >
> > Steve.
> >
> > On Wed, Feb 24, 2016 at 11:36 PM, Dan Smith <ds...@pivotal.io> wrote:
> >
> > > Hi Steve,
> > >
> > > I think your best bet is probably to use an OQL index. You can create
> an
> > > OQL index and then do range queries through OQL.
> > >
> > > You can see some examples of OQL queries here:
> > >
> > >
> > >
> >
> http://geode.docs.pivotal.io/docs/getting_started/querying_quick_reference.html#reference_D5CE64F5FD6F4A808AEFB748C867189E
> > >
> > > Eg
> > > SELECT * from /exampleRegion value WHERE value.ID > startKey AND
> > value.ID <
> > > endKey
> > >
> > > -Dan
> > >
> > > On Tue, Feb 23, 2016 at 10:37 PM, steve mathew <
> steve.mathew47@gmail.com
> > >
> > > wrote:
> > >
> > > > Hello I am new to the apache geode and exploring a possibility to
> store
> > > > data sorted into regions.
> > > >
> > > > I need to store data sorted for the scan(startkey, endkey) operation.
> > > > Simple looking at the APIs it looks geode is not designed for this.
> Is
> > > > there any way to achieve this?
> > > >
> > > > Thanks in advance.
> > > >
> > > > Kindly advise.
> > > >
> > > > -Steve.
> > > >
> > >
> >
>
Re: geode for specific usecase
Posted by Michael Stolz <ms...@pivotal.io>.
If you are storing the data using pdx auto seriaization the index will work
just fine with your custom object keys.
There is no short answer to how much extra memory overhead an index will
cause because it is dependent on the actual data itself (really the
cardinality of the data). See
https://en.wikipedia.org/wiki/Cardinality_(SQL_statements)
They do tend to be large. Use indices judiciously.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: 631-835-4771
On Wed, Feb 24, 2016 at 12:34 PM, steve mathew <st...@gmail.com>
wrote:
> Thank you Dan.
>
> Will this work with custom object keys?
> How much will be extra memory overhead when creating index?
>
> Steve.
>
> On Wed, Feb 24, 2016 at 11:36 PM, Dan Smith <ds...@pivotal.io> wrote:
>
> > Hi Steve,
> >
> > I think your best bet is probably to use an OQL index. You can create an
> > OQL index and then do range queries through OQL.
> >
> > You can see some examples of OQL queries here:
> >
> >
> >
> http://geode.docs.pivotal.io/docs/getting_started/querying_quick_reference.html#reference_D5CE64F5FD6F4A808AEFB748C867189E
> >
> > Eg
> > SELECT * from /exampleRegion value WHERE value.ID > startKey AND
> value.ID <
> > endKey
> >
> > -Dan
> >
> > On Tue, Feb 23, 2016 at 10:37 PM, steve mathew <steve.mathew47@gmail.com
> >
> > wrote:
> >
> > > Hello I am new to the apache geode and exploring a possibility to store
> > > data sorted into regions.
> > >
> > > I need to store data sorted for the scan(startkey, endkey) operation.
> > > Simple looking at the APIs it looks geode is not designed for this. Is
> > > there any way to achieve this?
> > >
> > > Thanks in advance.
> > >
> > > Kindly advise.
> > >
> > > -Steve.
> > >
> >
>
Re: geode for specific usecase
Posted by steve mathew <st...@gmail.com>.
Thank you Dan.
Will this work with custom object keys?
How much will be extra memory overhead when creating index?
Steve.
On Wed, Feb 24, 2016 at 11:36 PM, Dan Smith <ds...@pivotal.io> wrote:
> Hi Steve,
>
> I think your best bet is probably to use an OQL index. You can create an
> OQL index and then do range queries through OQL.
>
> You can see some examples of OQL queries here:
>
>
> http://geode.docs.pivotal.io/docs/getting_started/querying_quick_reference.html#reference_D5CE64F5FD6F4A808AEFB748C867189E
>
> Eg
> SELECT * from /exampleRegion value WHERE value.ID > startKey AND value.ID <
> endKey
>
> -Dan
>
> On Tue, Feb 23, 2016 at 10:37 PM, steve mathew <st...@gmail.com>
> wrote:
>
> > Hello I am new to the apache geode and exploring a possibility to store
> > data sorted into regions.
> >
> > I need to store data sorted for the scan(startkey, endkey) operation.
> > Simple looking at the APIs it looks geode is not designed for this. Is
> > there any way to achieve this?
> >
> > Thanks in advance.
> >
> > Kindly advise.
> >
> > -Steve.
> >
>
Re: geode for specific usecase
Posted by Dan Smith <ds...@pivotal.io>.
Hi Steve,
I think your best bet is probably to use an OQL index. You can create an
OQL index and then do range queries through OQL.
You can see some examples of OQL queries here:
http://geode.docs.pivotal.io/docs/getting_started/querying_quick_reference.html#reference_D5CE64F5FD6F4A808AEFB748C867189E
Eg
SELECT * from /exampleRegion value WHERE value.ID > startKey AND value.ID <
endKey
-Dan
On Tue, Feb 23, 2016 at 10:37 PM, steve mathew <st...@gmail.com>
wrote:
> Hello I am new to the apache geode and exploring a possibility to store
> data sorted into regions.
>
> I need to store data sorted for the scan(startkey, endkey) operation.
> Simple looking at the APIs it looks geode is not designed for this. Is
> there any way to achieve this?
>
> Thanks in advance.
>
> Kindly advise.
>
> -Steve.
>