You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Uwe Reh <re...@hebis.uni-frankfurt.de> on 2015/10/08 12:24:16 UTC

Re: faceting is unusable slow since upgrade to 5.3.0

Sorry for the delay. I had an ugly flu.

SOLR-7730 seems to work fine. Using docValues with Solr 
5.4.0-2015-09-29_08-29-55 1705813 makes my faceted queries fast again. 
(90ms vs. 20000ms) :-)

Thanks
Uwe



Am 27.09.2015 um 20:32 schrieb Mikhail Khludnev:
> On Sun, Sep 27, 2015 at 2:00 PM, Uwe Reh <re...@hebis.uni-frankfurt.de> wrote:
>
>> When 5.4 with SOLR-7730 will be released, I will start to use docValues.
>> Going this way, seems more straight forward to me.
>
>
> Sure. Giving your answers docValues facets has a really good chance to
> perform in your index after SOLR-7730. It's really interesting to see
> performance numbers on early 5.4 builds:
> https://builds.apache.org/view/All/job/Solr-Artifacts-5.x/lastSuccessfulBuild/artifact/solr/package/
>


Re: faceting is unusable slow since upgrade to 5.3.0

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
This fix definitely help for facet.field over docvalues field on
mult-segment index since 5.4.
I suppose it's irrelevant to JSON Facets, non-dv field, and pre 5.4.
I can not comment about comparing perfomance of dv and non-dv fields,
because "it depends" (с) benchmarking and profiler are the only advisers.

On Thu, Dec 17, 2015 at 9:22 AM, William Bell <bi...@gmail.com> wrote:

> Same question here....
>
> Wondering if faceting performance is fixed and how to take advantage of it
> ?
>
> On Wed, Dec 16, 2015 at 2:57 AM, Vincenzo D'Amore <v....@gmail.com>
> wrote:
>
> > Hi all,
> >
> > given that solr 5.4 is finally released, is this what's more stable and
> > efficient version of solrcloud ?
> >
> > I have a website which receives many search requests. It serve normally
> > about 2000 concurrent requests, but sometime there are peak from 4000 to
> > 10000 requests in few seconds.
> >
> > On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster
> to
> > a new brand version, but following this thread I read about the problems
> > that can occur upgrading to latest version.
> >
> > I have seen that issue SOLR-7730 "speed-up faceting on doc values fields"
> > is fixed in 5.4.
> >
> > I'm using standard faceting without docValues. Should I add docValues in
> > order to benefit of such fix?
> >
> > Best regards,
> > Vincenzo
> >
> >
> >
> > On Thu, Oct 8, 2015 at 2:22 PM, Mikhail Khludnev <
> > mkhludnev@griddynamics.com
> > > wrote:
> >
> > > Uwe, it's good to know! I mean that you've recovered. Take care!
> > >
> > > On Thu, Oct 8, 2015 at 1:24 PM, Uwe Reh <re...@hebis.uni-frankfurt.de>
> > > wrote:
> > >
> > > > Sorry for the delay. I had an ugly flu.
> > > >
> > > > SOLR-7730 seems to work fine. Using docValues with Solr
> > > > 5.4.0-2015-09-29_08-29-55 1705813 makes my faceted queries fast
> again.
> > > > (90ms vs. 20000ms) :-)
> > > >
> > > > Thanks
> > > > Uwe
> > > >
> > > >
> > > >
> > > >
> > > > Am 27.09.2015 um 20:32 schrieb Mikhail Khludnev:
> > > >
> > > >> On Sun, Sep 27, 2015 at 2:00 PM, Uwe Reh <
> reh@hebis.uni-frankfurt.de>
> > > >> wrote:
> > > >>
> > > >> When 5.4 with SOLR-7730 will be released, I will start to use
> > docValues.
> > > >>> Going this way, seems more straight forward to me.
> > > >>>
> > > >>
> > > >>
> > > >> Sure. Giving your answers docValues facets has a really good chance
> to
> > > >> perform in your index after SOLR-7730. It's really interesting to
> see
> > > >> performance numbers on early 5.4 builds:
> > > >>
> > > >>
> > >
> >
> https://builds.apache.org/view/All/job/Solr-Artifacts-5.x/lastSuccessfulBuild/artifact/solr/package/
> > > >>
> > > >>
> > > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > > Principal Engineer,
> > > Grid Dynamics
> > >
> > > <http://www.griddynamics.com>
> > > <mk...@griddynamics.com>
> > >
> >
> >
> >
> > --
> > Vincenzo D'Amore
> > email: v.damore@gmail.com
> > skype: free.dev
> > mobile: +39 349 8513251
> >
>
>
>
> --
> Bill Bell
> billnbell@gmail.com
> cell 720-256-8076
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mk...@griddynamics.com>

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by William Bell <bi...@gmail.com>.
Same question here....

Wondering if faceting performance is fixed and how to take advantage of it ?

On Wed, Dec 16, 2015 at 2:57 AM, Vincenzo D'Amore <v....@gmail.com>
wrote:

> Hi all,
>
> given that solr 5.4 is finally released, is this what's more stable and
> efficient version of solrcloud ?
>
> I have a website which receives many search requests. It serve normally
> about 2000 concurrent requests, but sometime there are peak from 4000 to
> 10000 requests in few seconds.
>
> On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster to
> a new brand version, but following this thread I read about the problems
> that can occur upgrading to latest version.
>
> I have seen that issue SOLR-7730 "speed-up faceting on doc values fields"
> is fixed in 5.4.
>
> I'm using standard faceting without docValues. Should I add docValues in
> order to benefit of such fix?
>
> Best regards,
> Vincenzo
>
>
>
> On Thu, Oct 8, 2015 at 2:22 PM, Mikhail Khludnev <
> mkhludnev@griddynamics.com
> > wrote:
>
> > Uwe, it's good to know! I mean that you've recovered. Take care!
> >
> > On Thu, Oct 8, 2015 at 1:24 PM, Uwe Reh <re...@hebis.uni-frankfurt.de>
> > wrote:
> >
> > > Sorry for the delay. I had an ugly flu.
> > >
> > > SOLR-7730 seems to work fine. Using docValues with Solr
> > > 5.4.0-2015-09-29_08-29-55 1705813 makes my faceted queries fast again.
> > > (90ms vs. 20000ms) :-)
> > >
> > > Thanks
> > > Uwe
> > >
> > >
> > >
> > >
> > > Am 27.09.2015 um 20:32 schrieb Mikhail Khludnev:
> > >
> > >> On Sun, Sep 27, 2015 at 2:00 PM, Uwe Reh <re...@hebis.uni-frankfurt.de>
> > >> wrote:
> > >>
> > >> When 5.4 with SOLR-7730 will be released, I will start to use
> docValues.
> > >>> Going this way, seems more straight forward to me.
> > >>>
> > >>
> > >>
> > >> Sure. Giving your answers docValues facets has a really good chance to
> > >> perform in your index after SOLR-7730. It's really interesting to see
> > >> performance numbers on early 5.4 builds:
> > >>
> > >>
> >
> https://builds.apache.org/view/All/job/Solr-Artifacts-5.x/lastSuccessfulBuild/artifact/solr/package/
> > >>
> > >>
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> > <mk...@griddynamics.com>
> >
>
>
>
> --
> Vincenzo D'Amore
> email: v.damore@gmail.com
> skype: free.dev
> mobile: +39 349 8513251
>



-- 
Bill Bell
billnbell@gmail.com
cell 720-256-8076

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by Jamie Johnson <je...@gmail.com>.
Also can we get the capability to choose the method of faceting in the
older faceting component?  I'm not looking for complete feature parity just
the ability to specify the method.  As always thanks.

On Fri, Dec 18, 2015 at 8:04 AM, Jamie Johnson <je...@gmail.com> wrote:

> Can we still specify the cache implementation for the field cache?  When
> this change occurred to faceting (uninverting reader vs field ) it
> prevented us from moving to 5.x but if we can get the 4.x functionality
> using that api we could look to port to the latest.
>
> Jamie
> On Dec 17, 2015 9:18 AM, "Yonik Seeley" <ys...@gmail.com> wrote:
>
>> On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore <v....@gmail.com>
>> wrote:
>> > Hi all,
>> >
>> > given that solr 5.4 is finally released, is this what's more stable and
>> > efficient version of solrcloud ?
>> >
>> > I have a website which receives many search requests. It serve normally
>> > about 2000 concurrent requests, but sometime there are peak from 4000 to
>> > 10000 requests in few seconds.
>> >
>> > On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster
>> to
>> > a new brand version, but following this thread I read about the problems
>> > that can occur upgrading to latest version.
>> >
>> > I have seen that issue SOLR-7730 "speed-up faceting on doc values
>> fields"
>> > is fixed in 5.4.
>> >
>> > I'm using standard faceting without docValues. Should I add docValues in
>> > order to benefit of such fix?
>>
>> You'll have to try it I think...
>> DocValues have a lot of advantages (much less heap consumption, and
>> much smaller overhead when opening a new searcher), but they can often
>> be slower as well.
>>
>> Comparing 4x to 5x non-docvalues, top-level field caches were removed
>> by lucene, and while that benefits certain things like NRT (opening a
>> new searcher very often), it will hurt performance for other
>> configurations.
>>
>> The JSON Facet API currently allows you to pick your strategy via the
>> "method" param for multi-valued string fields without docvalues:
>> "uif" (UninvertedField) gets you the top-level strategy from Solr 4,
>> while "dv" (DocValues built on-the-fly) gets you the NRT-friendly
>> "per-segment" strategy.
>>
>> -Yonik
>>
>

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by Jamie Johnson <je...@gmail.com>.
Can we still specify the cache implementation for the field cache?  When
this change occurred to faceting (uninverting reader vs field ) it
prevented us from moving to 5.x but if we can get the 4.x functionality
using that api we could look to port to the latest.

Jamie
On Dec 17, 2015 9:18 AM, "Yonik Seeley" <ys...@gmail.com> wrote:

> On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore <v....@gmail.com>
> wrote:
> > Hi all,
> >
> > given that solr 5.4 is finally released, is this what's more stable and
> > efficient version of solrcloud ?
> >
> > I have a website which receives many search requests. It serve normally
> > about 2000 concurrent requests, but sometime there are peak from 4000 to
> > 10000 requests in few seconds.
> >
> > On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster
> to
> > a new brand version, but following this thread I read about the problems
> > that can occur upgrading to latest version.
> >
> > I have seen that issue SOLR-7730 "speed-up faceting on doc values fields"
> > is fixed in 5.4.
> >
> > I'm using standard faceting without docValues. Should I add docValues in
> > order to benefit of such fix?
>
> You'll have to try it I think...
> DocValues have a lot of advantages (much less heap consumption, and
> much smaller overhead when opening a new searcher), but they can often
> be slower as well.
>
> Comparing 4x to 5x non-docvalues, top-level field caches were removed
> by lucene, and while that benefits certain things like NRT (opening a
> new searcher very often), it will hurt performance for other
> configurations.
>
> The JSON Facet API currently allows you to pick your strategy via the
> "method" param for multi-valued string fields without docvalues:
> "uif" (UninvertedField) gets you the top-level strategy from Solr 4,
> while "dv" (DocValues built on-the-fly) gets you the NRT-friendly
> "per-segment" strategy.
>
> -Yonik
>

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by William Bell <bi...@gmail.com>.
Thanks Jamie.

On Sat, Dec 19, 2015 at 11:31 PM, Jamie Johnson <je...@gmail.com> wrote:

> Bill,
>
> Check out the patch attached to
> https://issues.apache.org/jira/browse/SOLR-8096.  I had considered making
> the method uif after I had done most of the work, it would be trivial to
> change and would probably be more aligned with not adding unexpected
> changes to people that are currently using fc.
>
> -Jamie
>
> On Sat, Dec 19, 2015 at 11:03 PM, William Bell <bi...@gmail.com>
> wrote:
>
> > Can we add method=uif back when not using the JSON Facet API too?
> >
> > That would help a lot of people.
> >
> > On Thu, Dec 17, 2015 at 7:17 AM, Yonik Seeley <ys...@gmail.com> wrote:
> >
> > > On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore <v....@gmail.com>
> > > wrote:
> > > > Hi all,
> > > >
> > > > given that solr 5.4 is finally released, is this what's more stable
> and
> > > > efficient version of solrcloud ?
> > > >
> > > > I have a website which receives many search requests. It serve
> normally
> > > > about 2000 concurrent requests, but sometime there are peak from 4000
> > to
> > > > 10000 requests in few seconds.
> > > >
> > > > On January I'll have a chance to upgrade my old SolrCloud 4.8.1
> cluster
> > > to
> > > > a new brand version, but following this thread I read about the
> > problems
> > > > that can occur upgrading to latest version.
> > > >
> > > > I have seen that issue SOLR-7730 "speed-up faceting on doc values
> > fields"
> > > > is fixed in 5.4.
> > > >
> > > > I'm using standard faceting without docValues. Should I add docValues
> > in
> > > > order to benefit of such fix?
> > >
> > > You'll have to try it I think...
> > > DocValues have a lot of advantages (much less heap consumption, and
> > > much smaller overhead when opening a new searcher), but they can often
> > > be slower as well.
> > >
> > > Comparing 4x to 5x non-docvalues, top-level field caches were removed
> > > by lucene, and while that benefits certain things like NRT (opening a
> > > new searcher very often), it will hurt performance for other
> > > configurations.
> > >
> > > The JSON Facet API currently allows you to pick your strategy via the
> > > "method" param for multi-valued string fields without docvalues:
> > > "uif" (UninvertedField) gets you the top-level strategy from Solr 4,
> > > while "dv" (DocValues built on-the-fly) gets you the NRT-friendly
> > > "per-segment" strategy.
> > >
> > > -Yonik
> > >
> >
> >
> >
> > --
> > Bill Bell
> > billnbell@gmail.com
> > cell 720-256-8076
> >
>



-- 
Bill Bell
billnbell@gmail.com
cell 720-256-8076

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by Jamie Johnson <je...@gmail.com>.
Bill,

Check out the patch attached to
https://issues.apache.org/jira/browse/SOLR-8096.  I had considered making
the method uif after I had done most of the work, it would be trivial to
change and would probably be more aligned with not adding unexpected
changes to people that are currently using fc.

-Jamie

On Sat, Dec 19, 2015 at 11:03 PM, William Bell <bi...@gmail.com> wrote:

> Can we add method=uif back when not using the JSON Facet API too?
>
> That would help a lot of people.
>
> On Thu, Dec 17, 2015 at 7:17 AM, Yonik Seeley <ys...@gmail.com> wrote:
>
> > On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore <v....@gmail.com>
> > wrote:
> > > Hi all,
> > >
> > > given that solr 5.4 is finally released, is this what's more stable and
> > > efficient version of solrcloud ?
> > >
> > > I have a website which receives many search requests. It serve normally
> > > about 2000 concurrent requests, but sometime there are peak from 4000
> to
> > > 10000 requests in few seconds.
> > >
> > > On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster
> > to
> > > a new brand version, but following this thread I read about the
> problems
> > > that can occur upgrading to latest version.
> > >
> > > I have seen that issue SOLR-7730 "speed-up faceting on doc values
> fields"
> > > is fixed in 5.4.
> > >
> > > I'm using standard faceting without docValues. Should I add docValues
> in
> > > order to benefit of such fix?
> >
> > You'll have to try it I think...
> > DocValues have a lot of advantages (much less heap consumption, and
> > much smaller overhead when opening a new searcher), but they can often
> > be slower as well.
> >
> > Comparing 4x to 5x non-docvalues, top-level field caches were removed
> > by lucene, and while that benefits certain things like NRT (opening a
> > new searcher very often), it will hurt performance for other
> > configurations.
> >
> > The JSON Facet API currently allows you to pick your strategy via the
> > "method" param for multi-valued string fields without docvalues:
> > "uif" (UninvertedField) gets you the top-level strategy from Solr 4,
> > while "dv" (DocValues built on-the-fly) gets you the NRT-friendly
> > "per-segment" strategy.
> >
> > -Yonik
> >
>
>
>
> --
> Bill Bell
> billnbell@gmail.com
> cell 720-256-8076
>

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by William Bell <bi...@gmail.com>.
Can we add method=uif back when not using the JSON Facet API too?

That would help a lot of people.

On Thu, Dec 17, 2015 at 7:17 AM, Yonik Seeley <ys...@gmail.com> wrote:

> On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore <v....@gmail.com>
> wrote:
> > Hi all,
> >
> > given that solr 5.4 is finally released, is this what's more stable and
> > efficient version of solrcloud ?
> >
> > I have a website which receives many search requests. It serve normally
> > about 2000 concurrent requests, but sometime there are peak from 4000 to
> > 10000 requests in few seconds.
> >
> > On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster
> to
> > a new brand version, but following this thread I read about the problems
> > that can occur upgrading to latest version.
> >
> > I have seen that issue SOLR-7730 "speed-up faceting on doc values fields"
> > is fixed in 5.4.
> >
> > I'm using standard faceting without docValues. Should I add docValues in
> > order to benefit of such fix?
>
> You'll have to try it I think...
> DocValues have a lot of advantages (much less heap consumption, and
> much smaller overhead when opening a new searcher), but they can often
> be slower as well.
>
> Comparing 4x to 5x non-docvalues, top-level field caches were removed
> by lucene, and while that benefits certain things like NRT (opening a
> new searcher very often), it will hurt performance for other
> configurations.
>
> The JSON Facet API currently allows you to pick your strategy via the
> "method" param for multi-valued string fields without docvalues:
> "uif" (UninvertedField) gets you the top-level strategy from Solr 4,
> while "dv" (DocValues built on-the-fly) gets you the NRT-friendly
> "per-segment" strategy.
>
> -Yonik
>



-- 
Bill Bell
billnbell@gmail.com
cell 720-256-8076

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by Yonik Seeley <ys...@gmail.com>.
On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore <v....@gmail.com> wrote:
> Hi all,
>
> given that solr 5.4 is finally released, is this what's more stable and
> efficient version of solrcloud ?
>
> I have a website which receives many search requests. It serve normally
> about 2000 concurrent requests, but sometime there are peak from 4000 to
> 10000 requests in few seconds.
>
> On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster to
> a new brand version, but following this thread I read about the problems
> that can occur upgrading to latest version.
>
> I have seen that issue SOLR-7730 "speed-up faceting on doc values fields"
> is fixed in 5.4.
>
> I'm using standard faceting without docValues. Should I add docValues in
> order to benefit of such fix?

You'll have to try it I think...
DocValues have a lot of advantages (much less heap consumption, and
much smaller overhead when opening a new searcher), but they can often
be slower as well.

Comparing 4x to 5x non-docvalues, top-level field caches were removed
by lucene, and while that benefits certain things like NRT (opening a
new searcher very often), it will hurt performance for other
configurations.

The JSON Facet API currently allows you to pick your strategy via the
"method" param for multi-valued string fields without docvalues:
"uif" (UninvertedField) gets you the top-level strategy from Solr 4,
while "dv" (DocValues built on-the-fly) gets you the NRT-friendly
"per-segment" strategy.

-Yonik

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by Vincenzo D'Amore <v....@gmail.com>.
Hi all,

given that solr 5.4 is finally released, is this what's more stable and
efficient version of solrcloud ?

I have a website which receives many search requests. It serve normally
about 2000 concurrent requests, but sometime there are peak from 4000 to
10000 requests in few seconds.

On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster to
a new brand version, but following this thread I read about the problems
that can occur upgrading to latest version.

I have seen that issue SOLR-7730 "speed-up faceting on doc values fields"
is fixed in 5.4.

I'm using standard faceting without docValues. Should I add docValues in
order to benefit of such fix?

Best regards,
Vincenzo



On Thu, Oct 8, 2015 at 2:22 PM, Mikhail Khludnev <mkhludnev@griddynamics.com
> wrote:

> Uwe, it's good to know! I mean that you've recovered. Take care!
>
> On Thu, Oct 8, 2015 at 1:24 PM, Uwe Reh <re...@hebis.uni-frankfurt.de>
> wrote:
>
> > Sorry for the delay. I had an ugly flu.
> >
> > SOLR-7730 seems to work fine. Using docValues with Solr
> > 5.4.0-2015-09-29_08-29-55 1705813 makes my faceted queries fast again.
> > (90ms vs. 20000ms) :-)
> >
> > Thanks
> > Uwe
> >
> >
> >
> >
> > Am 27.09.2015 um 20:32 schrieb Mikhail Khludnev:
> >
> >> On Sun, Sep 27, 2015 at 2:00 PM, Uwe Reh <re...@hebis.uni-frankfurt.de>
> >> wrote:
> >>
> >> When 5.4 with SOLR-7730 will be released, I will start to use docValues.
> >>> Going this way, seems more straight forward to me.
> >>>
> >>
> >>
> >> Sure. Giving your answers docValues facets has a really good chance to
> >> perform in your index after SOLR-7730. It's really interesting to see
> >> performance numbers on early 5.4 builds:
> >>
> >>
> https://builds.apache.org/view/All/job/Solr-Artifacts-5.x/lastSuccessfulBuild/artifact/solr/package/
> >>
> >>
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>



-- 
Vincenzo D'Amore
email: v.damore@gmail.com
skype: free.dev
mobile: +39 349 8513251

Re: faceting is unusable slow since upgrade to 5.3.0

Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Uwe, it's good to know! I mean that you've recovered. Take care!

On Thu, Oct 8, 2015 at 1:24 PM, Uwe Reh <re...@hebis.uni-frankfurt.de> wrote:

> Sorry for the delay. I had an ugly flu.
>
> SOLR-7730 seems to work fine. Using docValues with Solr
> 5.4.0-2015-09-29_08-29-55 1705813 makes my faceted queries fast again.
> (90ms vs. 20000ms) :-)
>
> Thanks
> Uwe
>
>
>
>
> Am 27.09.2015 um 20:32 schrieb Mikhail Khludnev:
>
>> On Sun, Sep 27, 2015 at 2:00 PM, Uwe Reh <re...@hebis.uni-frankfurt.de>
>> wrote:
>>
>> When 5.4 with SOLR-7730 will be released, I will start to use docValues.
>>> Going this way, seems more straight forward to me.
>>>
>>
>>
>> Sure. Giving your answers docValues facets has a really good chance to
>> perform in your index after SOLR-7730. It's really interesting to see
>> performance numbers on early 5.4 builds:
>>
>> https://builds.apache.org/view/All/job/Solr-Artifacts-5.x/lastSuccessfulBuild/artifact/solr/package/
>>
>>
>


-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mk...@griddynamics.com>