You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by javadevmtl <ja...@gmail.com> on 2015/08/12 22:21:09 UTC

Multiple indexes.

Using 1.3.0

I'm testing insertion performance. My model requires 12 secondary indexes.
I.e: 12 x @QuerySqlField(index = true)

At 6 indexes (meaning I comment out 6 of the 12 indexes)
I can get about 11,000 inserts per second with avg 1ms (includes jmeter
latency) using data streamer and tested up to 14 million records.

At 8 indexes
I can get about 7,900 inserts per second with avg 8ms (includes jmeter
latency) using data streamer and tested up to 14 million records.

At 12 indexes it starts at 11K and slowly degrades to 5,000 and less inserts
per second and of course higher latency.

JMeter always send the same amount of data.

Just wondering if it worth separating my model into separate objects and do
2 inserts to 2 separate caches and do cross cache joins.






--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Multiple indexes.

Posted by Konstantin Boudnik <co...@apache.org>.
Oh better yet - open a JIRA and contribute a patch, and improve your karma;)

On August 13, 2015 1:41:57 PM PDT, Dmitriy Setrakyan <ds...@apache.org> wrote:
>@javadevmtl - Do you have a test we can use to reproduce it? You can
>either
>provide a github repo link or paste your code into a Gist or Pastebin.
>
>D.
>
>On Thu, Aug 13, 2015 at 1:37 PM, Sergi Vladykin
><se...@gmail.com>
>wrote:
>
>> Yes, this can be related to serialization. I think we should test
>this
>> ourselves to make sure.
>> Igniters, can someone add such a benchmark to our performance suite
>and
>> check what happens?
>>
>> Sergi
>>
>> 2015-08-13 17:24 GMT+03:00 javadevmtl <ja...@gmail.com>:
>>
>>> I think I know what it is...
>>>
>>> It's not GC everything is off-heap and I start VM with XMX4g which
>is more
>>> then enough heap to handle the incoming HTTP requests parse the
>json,
>>> convert to object and serialize in the off-heap cache.
>>>
>>> Basically, the model has 12 fields. 2 of those fields are "home"
>address
>>> fields e.g: 1234 Street, Apt: 9, New York... And they can be up to
>240
>>> characters, so 480 characters total.
>>>
>>> I removed those 2 fields for now and I can index 10 fields (most of
>which
>>> are still strings of 60 chars) at 11K requests per second with no
>>> slowdown.
>>>
>>> So the 2 fields either 2 long for an index or affecting
>serialization to
>>> the
>>> cache???
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>>
>http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936p945.html
>>> Sent from the Apache Ignite Users mailing list archive at
>Nabble.com.
>>>
>>
>>


Re: Multiple indexes.

Posted by Dmitriy Setrakyan <ds...@apache.org>.
@javadevmtl - Do you have a test we can use to reproduce it? You can either
provide a github repo link or paste your code into a Gist or Pastebin.

D.

On Thu, Aug 13, 2015 at 1:37 PM, Sergi Vladykin <se...@gmail.com>
wrote:

> Yes, this can be related to serialization. I think we should test this
> ourselves to make sure.
> Igniters, can someone add such a benchmark to our performance suite and
> check what happens?
>
> Sergi
>
> 2015-08-13 17:24 GMT+03:00 javadevmtl <ja...@gmail.com>:
>
>> I think I know what it is...
>>
>> It's not GC everything is off-heap and I start VM with XMX4g which is more
>> then enough heap to handle the incoming HTTP requests parse the json,
>> convert to object and serialize in the off-heap cache.
>>
>> Basically, the model has 12 fields. 2 of those fields are "home" address
>> fields e.g: 1234 Street, Apt: 9, New York... And they can be up to 240
>> characters, so 480 characters total.
>>
>> I removed those 2 fields for now and I can index 10 fields (most of which
>> are still strings of 60 chars) at 11K requests per second with no
>> slowdown.
>>
>> So the 2 fields either 2 long for an index or affecting serialization to
>> the
>> cache???
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936p945.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Multiple indexes.

Posted by Sergi Vladykin <se...@gmail.com>.
Yes, this can be related to serialization. I think we should test this
ourselves to make sure.
Igniters, can someone add such a benchmark to our performance suite and
check what happens?

Sergi

2015-08-13 17:24 GMT+03:00 javadevmtl <ja...@gmail.com>:

> I think I know what it is...
>
> It's not GC everything is off-heap and I start VM with XMX4g which is more
> then enough heap to handle the incoming HTTP requests parse the json,
> convert to object and serialize in the off-heap cache.
>
> Basically, the model has 12 fields. 2 of those fields are "home" address
> fields e.g: 1234 Street, Apt: 9, New York... And they can be up to 240
> characters, so 480 characters total.
>
> I removed those 2 fields for now and I can index 10 fields (most of which
> are still strings of 60 chars) at 11K requests per second with no slowdown.
>
> So the 2 fields either 2 long for an index or affecting serialization to
> the
> cache???
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936p945.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Multiple indexes.

Posted by javadevmtl <ja...@gmail.com>.
I think I know what it is...

It's not GC everything is off-heap and I start VM with XMX4g which is more
then enough heap to handle the incoming HTTP requests parse the json,
convert to object and serialize in the off-heap cache.

Basically, the model has 12 fields. 2 of those fields are "home" address
fields e.g: 1234 Street, Apt: 9, New York... And they can be up to 240
characters, so 480 characters total.

I removed those 2 fields for now and I can index 10 fields (most of which
are still strings of 60 chars) at 11K requests per second with no slowdown.

So the 2 fields either 2 long for an index or affecting serialization to the
cache???







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936p945.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Multiple indexes.

Posted by vkulichenko <va...@gmail.com>.
javadevmtl wrote
> Just wondering if it worth separating my model into separate objects and
> do 2 inserts to 2 separate caches and do cross cache joins.

Personally I don't think that such separation can affect performance, but
you can try and see if it changes.

javadevmtl wrote
> At 12 indexes it starts at 11K and slowly degrades to 5,000 and less
> inserts per second and of course higher latency.

This looks like a GC issue. Please make sure that you have enough memory to
store the data.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Multiple-indexes-tp936p938.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.