You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Martynas L <ma...@gmail.com> on 2020/12/03 14:04:04 UTC

Slower document retrieval in 8.7.0 comparing to 7.5.0

Hi,
We've migrated from 7.5.0 to 8.7.0 and find out that the index "searching"
is significantly (4-5 times) slower in the latest version.
It seems that
org.apache.lucene.search.IndexSearcher#doc(int)
is slower.

Is it possible to have similar performance with 8.7.0?

Best regards,
Martynas

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Martynas L <ma...@gmail.com>.
The accent should not be on retrieved documents number, but on the duration
ratio - 8.7.0 is 3 times slower. I think it will be similar ratio
retrieving any number of documents.

On Fri, Jan 22, 2021 at 1:39 PM Rob Audenaerde <ro...@gmail.com>
wrote:

> Hi Martrynas,
>
> In your sample code you are retrieving all (1 million!) documents from the
> index, that surely is not a good match for lucene  :)
>
> Is that a good reflection of your use-case?
>
> On Fri, Jan 22, 2021 at 9:52 AM Martynas L <ma...@gmail.com> wrote:
>
> >  Please see the sample at
> > https://drive.google.com/drive/folders/1ufVZXzkugBAFnuy8HLAY6mbPWzjknrfE
> >
> > IndexGenerator - creates a dummy index.
> > IndexReader - retrieves documents - duration time with 7.5.0 version is
> > ~2s, while ~6s with 8.7.0
> >
> > Regards,
> > Martynas
> >
> >
> > On Thu, Jan 21, 2021 at 8:21 PM Rob Audenaerde <rob.audenaerde@gmail.com
> >
> > wrote:
> >
> > > There is no attachment in the previous email that I can see? Maybe you
> > can
> > > post it online?
> > >
> > > On Thu, Jan 21, 2021 at 4:54 PM Martynas L <ma...@gmail.com>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > Are there any comments on this issue?
> > > > If there is no workaround, we will be forced to rollback to the 7.5.0
> > > > version.
> > > >
> > > > Best regards,
> > > > Martynas
> > > >
> > > > On Tue, Jan 12, 2021 at 12:27 PM Martynas L <ma...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Please see attached sample.
> > > > > IndexGenerator - creates a dummy index.
> > > > > IndexReader - retrieves documents - duration time with 7.5.0
> version
> > is
> > > > > ~2s, while ~6s with 8.7.0
> > > > >
> > > > > Regards,
> > > > > Martynas
> > > > >
> > > > > On Tue, Dec 22, 2020 at 3:23 PM Vincenzo D'Amore <
> v.damore@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> I think it would be useful to have an example of a document and,
> if
> > > > >> possible, an example of query that takes too long.
> > > > >>
> > > > >> On Mon, Dec 21, 2020 at 1:47 PM Martynas L <
> martynas.sub@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hello,
> > > > >> >
> > > > >> > I am sorry for the delay.
> > > > >> >
> > > > >> > Not sure what you mean by "workload". We have a performance
> tests,
> > > > which
> > > > >> > started failing after upgrading to 8.7.0.
> > > > >> > So I just tried to query the index (built form the same source)
> to
> > > get
> > > > >> all
> > > > >> > documents and compare the performance with 7.5.0.
> > > > >> >
> > > > >> > Document "size" is a sum of all stored string lengths (3402519
> > > > >> documents):
> > > > >> >
> > > > >> > doc size 903 - 88s vs 22s
> > > > >> >
> > > > >> > doc size 36 (only one field loaded, used searcher.doc(docID,
> > > > >> > Collections.singleton("fieldName"))) - 78s vs 16s
> > > > >> >
> > > > >> > doc size 439 (some fields made not stored) - 46s vs 14.5s
> > > > >> >
> > > > >> > Best regards,
> > > > >> > Martynas
> > > > >> >
> > > > >> > On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jpountz@gmail.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > Hello Martynas,
> > > > >> > >
> > > > >> > > There have indeed been changes related to stored fields in
> 8.7.
> > > What
> > > > >> does
> > > > >> > > your workload look like and how large are your documents on
> > > average?
> > > > >> > >
> > > > >> > > On Thu, Dec 3, 2020 at 3:04 PM Martynas L <
> > martynas.sub@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > Hi,
> > > > >> > > > We've migrated from 7.5.0 to 8.7.0 and find out that the
> index
> > > > >> > > "searching"
> > > > >> > > > is significantly (4-5 times) slower in the latest version.
> > > > >> > > > It seems that
> > > > >> > > > org.apache.lucene.search.IndexSearcher#doc(int)
> > > > >> > > > is slower.
> > > > >> > > >
> > > > >> > > > Is it possible to have similar performance with 8.7.0?
> > > > >> > > >
> > > > >> > > > Best regards,
> > > > >> > > > Martynas
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > Adrien
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Vincenzo D'Amore
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Rob Audenaerde <ro...@gmail.com>.
Hi Martrynas,

In your sample code you are retrieving all (1 million!) documents from the
index, that surely is not a good match for lucene  :)

Is that a good reflection of your use-case?

On Fri, Jan 22, 2021 at 9:52 AM Martynas L <ma...@gmail.com> wrote:

>  Please see the sample at
> https://drive.google.com/drive/folders/1ufVZXzkugBAFnuy8HLAY6mbPWzjknrfE
>
> IndexGenerator - creates a dummy index.
> IndexReader - retrieves documents - duration time with 7.5.0 version is
> ~2s, while ~6s with 8.7.0
>
> Regards,
> Martynas
>
>
> On Thu, Jan 21, 2021 at 8:21 PM Rob Audenaerde <ro...@gmail.com>
> wrote:
>
> > There is no attachment in the previous email that I can see? Maybe you
> can
> > post it online?
> >
> > On Thu, Jan 21, 2021 at 4:54 PM Martynas L <ma...@gmail.com>
> wrote:
> >
> > > Hello,
> > >
> > > Are there any comments on this issue?
> > > If there is no workaround, we will be forced to rollback to the 7.5.0
> > > version.
> > >
> > > Best regards,
> > > Martynas
> > >
> > > On Tue, Jan 12, 2021 at 12:27 PM Martynas L <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Please see attached sample.
> > > > IndexGenerator - creates a dummy index.
> > > > IndexReader - retrieves documents - duration time with 7.5.0 version
> is
> > > > ~2s, while ~6s with 8.7.0
> > > >
> > > > Regards,
> > > > Martynas
> > > >
> > > > On Tue, Dec 22, 2020 at 3:23 PM Vincenzo D'Amore <v.damore@gmail.com
> >
> > > > wrote:
> > > >
> > > >> I think it would be useful to have an example of a document and, if
> > > >> possible, an example of query that takes too long.
> > > >>
> > > >> On Mon, Dec 21, 2020 at 1:47 PM Martynas L <ma...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hello,
> > > >> >
> > > >> > I am sorry for the delay.
> > > >> >
> > > >> > Not sure what you mean by "workload". We have a performance tests,
> > > which
> > > >> > started failing after upgrading to 8.7.0.
> > > >> > So I just tried to query the index (built form the same source) to
> > get
> > > >> all
> > > >> > documents and compare the performance with 7.5.0.
> > > >> >
> > > >> > Document "size" is a sum of all stored string lengths (3402519
> > > >> documents):
> > > >> >
> > > >> > doc size 903 - 88s vs 22s
> > > >> >
> > > >> > doc size 36 (only one field loaded, used searcher.doc(docID,
> > > >> > Collections.singleton("fieldName"))) - 78s vs 16s
> > > >> >
> > > >> > doc size 439 (some fields made not stored) - 46s vs 14.5s
> > > >> >
> > > >> > Best regards,
> > > >> > Martynas
> > > >> >
> > > >> > On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jp...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Hello Martynas,
> > > >> > >
> > > >> > > There have indeed been changes related to stored fields in 8.7.
> > What
> > > >> does
> > > >> > > your workload look like and how large are your documents on
> > average?
> > > >> > >
> > > >> > > On Thu, Dec 3, 2020 at 3:04 PM Martynas L <
> martynas.sub@gmail.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > Hi,
> > > >> > > > We've migrated from 7.5.0 to 8.7.0 and find out that the index
> > > >> > > "searching"
> > > >> > > > is significantly (4-5 times) slower in the latest version.
> > > >> > > > It seems that
> > > >> > > > org.apache.lucene.search.IndexSearcher#doc(int)
> > > >> > > > is slower.
> > > >> > > >
> > > >> > > > Is it possible to have similar performance with 8.7.0?
> > > >> > > >
> > > >> > > > Best regards,
> > > >> > > > Martynas
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Adrien
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Vincenzo D'Amore
> > > >>
> > > >
> > >
> >
>

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Martynas L <ma...@gmail.com>.
 Please see the sample at
https://drive.google.com/drive/folders/1ufVZXzkugBAFnuy8HLAY6mbPWzjknrfE

IndexGenerator - creates a dummy index.
IndexReader - retrieves documents - duration time with 7.5.0 version is
~2s, while ~6s with 8.7.0

Regards,
Martynas


On Thu, Jan 21, 2021 at 8:21 PM Rob Audenaerde <ro...@gmail.com>
wrote:

> There is no attachment in the previous email that I can see? Maybe you can
> post it online?
>
> On Thu, Jan 21, 2021 at 4:54 PM Martynas L <ma...@gmail.com> wrote:
>
> > Hello,
> >
> > Are there any comments on this issue?
> > If there is no workaround, we will be forced to rollback to the 7.5.0
> > version.
> >
> > Best regards,
> > Martynas
> >
> > On Tue, Jan 12, 2021 at 12:27 PM Martynas L <ma...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Please see attached sample.
> > > IndexGenerator - creates a dummy index.
> > > IndexReader - retrieves documents - duration time with 7.5.0 version is
> > > ~2s, while ~6s with 8.7.0
> > >
> > > Regards,
> > > Martynas
> > >
> > > On Tue, Dec 22, 2020 at 3:23 PM Vincenzo D'Amore <v....@gmail.com>
> > > wrote:
> > >
> > >> I think it would be useful to have an example of a document and, if
> > >> possible, an example of query that takes too long.
> > >>
> > >> On Mon, Dec 21, 2020 at 1:47 PM Martynas L <ma...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > I am sorry for the delay.
> > >> >
> > >> > Not sure what you mean by "workload". We have a performance tests,
> > which
> > >> > started failing after upgrading to 8.7.0.
> > >> > So I just tried to query the index (built form the same source) to
> get
> > >> all
> > >> > documents and compare the performance with 7.5.0.
> > >> >
> > >> > Document "size" is a sum of all stored string lengths (3402519
> > >> documents):
> > >> >
> > >> > doc size 903 - 88s vs 22s
> > >> >
> > >> > doc size 36 (only one field loaded, used searcher.doc(docID,
> > >> > Collections.singleton("fieldName"))) - 78s vs 16s
> > >> >
> > >> > doc size 439 (some fields made not stored) - 46s vs 14.5s
> > >> >
> > >> > Best regards,
> > >> > Martynas
> > >> >
> > >> > On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jp...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hello Martynas,
> > >> > >
> > >> > > There have indeed been changes related to stored fields in 8.7.
> What
> > >> does
> > >> > > your workload look like and how large are your documents on
> average?
> > >> > >
> > >> > > On Thu, Dec 3, 2020 at 3:04 PM Martynas L <martynas.sub@gmail.com
> >
> > >> > wrote:
> > >> > >
> > >> > > > Hi,
> > >> > > > We've migrated from 7.5.0 to 8.7.0 and find out that the index
> > >> > > "searching"
> > >> > > > is significantly (4-5 times) slower in the latest version.
> > >> > > > It seems that
> > >> > > > org.apache.lucene.search.IndexSearcher#doc(int)
> > >> > > > is slower.
> > >> > > >
> > >> > > > Is it possible to have similar performance with 8.7.0?
> > >> > > >
> > >> > > > Best regards,
> > >> > > > Martynas
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Adrien
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> Vincenzo D'Amore
> > >>
> > >
> >
>

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Rob Audenaerde <ro...@gmail.com>.
There is no attachment in the previous email that I can see? Maybe you can
post it online?

On Thu, Jan 21, 2021 at 4:54 PM Martynas L <ma...@gmail.com> wrote:

> Hello,
>
> Are there any comments on this issue?
> If there is no workaround, we will be forced to rollback to the 7.5.0
> version.
>
> Best regards,
> Martynas
>
> On Tue, Jan 12, 2021 at 12:27 PM Martynas L <ma...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Please see attached sample.
> > IndexGenerator - creates a dummy index.
> > IndexReader - retrieves documents - duration time with 7.5.0 version is
> > ~2s, while ~6s with 8.7.0
> >
> > Regards,
> > Martynas
> >
> > On Tue, Dec 22, 2020 at 3:23 PM Vincenzo D'Amore <v....@gmail.com>
> > wrote:
> >
> >> I think it would be useful to have an example of a document and, if
> >> possible, an example of query that takes too long.
> >>
> >> On Mon, Dec 21, 2020 at 1:47 PM Martynas L <ma...@gmail.com>
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I am sorry for the delay.
> >> >
> >> > Not sure what you mean by "workload". We have a performance tests,
> which
> >> > started failing after upgrading to 8.7.0.
> >> > So I just tried to query the index (built form the same source) to get
> >> all
> >> > documents and compare the performance with 7.5.0.
> >> >
> >> > Document "size" is a sum of all stored string lengths (3402519
> >> documents):
> >> >
> >> > doc size 903 - 88s vs 22s
> >> >
> >> > doc size 36 (only one field loaded, used searcher.doc(docID,
> >> > Collections.singleton("fieldName"))) - 78s vs 16s
> >> >
> >> > doc size 439 (some fields made not stored) - 46s vs 14.5s
> >> >
> >> > Best regards,
> >> > Martynas
> >> >
> >> > On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jp...@gmail.com>
> wrote:
> >> >
> >> > > Hello Martynas,
> >> > >
> >> > > There have indeed been changes related to stored fields in 8.7. What
> >> does
> >> > > your workload look like and how large are your documents on average?
> >> > >
> >> > > On Thu, Dec 3, 2020 at 3:04 PM Martynas L <ma...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Hi,
> >> > > > We've migrated from 7.5.0 to 8.7.0 and find out that the index
> >> > > "searching"
> >> > > > is significantly (4-5 times) slower in the latest version.
> >> > > > It seems that
> >> > > > org.apache.lucene.search.IndexSearcher#doc(int)
> >> > > > is slower.
> >> > > >
> >> > > > Is it possible to have similar performance with 8.7.0?
> >> > > >
> >> > > > Best regards,
> >> > > > Martynas
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Adrien
> >> > >
> >> >
> >>
> >>
> >> --
> >> Vincenzo D'Amore
> >>
> >
>

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Martynas L <ma...@gmail.com>.
Hello,

Are there any comments on this issue?
If there is no workaround, we will be forced to rollback to the 7.5.0
version.

Best regards,
Martynas

On Tue, Jan 12, 2021 at 12:27 PM Martynas L <ma...@gmail.com> wrote:

> Hi,
>
> Please see attached sample.
> IndexGenerator - creates a dummy index.
> IndexReader - retrieves documents - duration time with 7.5.0 version is
> ~2s, while ~6s with 8.7.0
>
> Regards,
> Martynas
>
> On Tue, Dec 22, 2020 at 3:23 PM Vincenzo D'Amore <v....@gmail.com>
> wrote:
>
>> I think it would be useful to have an example of a document and, if
>> possible, an example of query that takes too long.
>>
>> On Mon, Dec 21, 2020 at 1:47 PM Martynas L <ma...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I am sorry for the delay.
>> >
>> > Not sure what you mean by "workload". We have a performance tests, which
>> > started failing after upgrading to 8.7.0.
>> > So I just tried to query the index (built form the same source) to get
>> all
>> > documents and compare the performance with 7.5.0.
>> >
>> > Document "size" is a sum of all stored string lengths (3402519
>> documents):
>> >
>> > doc size 903 - 88s vs 22s
>> >
>> > doc size 36 (only one field loaded, used searcher.doc(docID,
>> > Collections.singleton("fieldName"))) - 78s vs 16s
>> >
>> > doc size 439 (some fields made not stored) - 46s vs 14.5s
>> >
>> > Best regards,
>> > Martynas
>> >
>> > On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jp...@gmail.com> wrote:
>> >
>> > > Hello Martynas,
>> > >
>> > > There have indeed been changes related to stored fields in 8.7. What
>> does
>> > > your workload look like and how large are your documents on average?
>> > >
>> > > On Thu, Dec 3, 2020 at 3:04 PM Martynas L <ma...@gmail.com>
>> > wrote:
>> > >
>> > > > Hi,
>> > > > We've migrated from 7.5.0 to 8.7.0 and find out that the index
>> > > "searching"
>> > > > is significantly (4-5 times) slower in the latest version.
>> > > > It seems that
>> > > > org.apache.lucene.search.IndexSearcher#doc(int)
>> > > > is slower.
>> > > >
>> > > > Is it possible to have similar performance with 8.7.0?
>> > > >
>> > > > Best regards,
>> > > > Martynas
>> > > >
>> > >
>> > >
>> > > --
>> > > Adrien
>> > >
>> >
>>
>>
>> --
>> Vincenzo D'Amore
>>
>

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Martynas L <ma...@gmail.com>.
Hi,

Please see attached sample.
IndexGenerator - creates a dummy index.
IndexReader - retrieves documents - duration time with 7.5.0 version is
~2s, while ~6s with 8.7.0

Regards,
Martynas

On Tue, Dec 22, 2020 at 3:23 PM Vincenzo D'Amore <v....@gmail.com> wrote:

> I think it would be useful to have an example of a document and, if
> possible, an example of query that takes too long.
>
> On Mon, Dec 21, 2020 at 1:47 PM Martynas L <ma...@gmail.com> wrote:
>
> > Hello,
> >
> > I am sorry for the delay.
> >
> > Not sure what you mean by "workload". We have a performance tests, which
> > started failing after upgrading to 8.7.0.
> > So I just tried to query the index (built form the same source) to get
> all
> > documents and compare the performance with 7.5.0.
> >
> > Document "size" is a sum of all stored string lengths (3402519
> documents):
> >
> > doc size 903 - 88s vs 22s
> >
> > doc size 36 (only one field loaded, used searcher.doc(docID,
> > Collections.singleton("fieldName"))) - 78s vs 16s
> >
> > doc size 439 (some fields made not stored) - 46s vs 14.5s
> >
> > Best regards,
> > Martynas
> >
> > On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jp...@gmail.com> wrote:
> >
> > > Hello Martynas,
> > >
> > > There have indeed been changes related to stored fields in 8.7. What
> does
> > > your workload look like and how large are your documents on average?
> > >
> > > On Thu, Dec 3, 2020 at 3:04 PM Martynas L <ma...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > > We've migrated from 7.5.0 to 8.7.0 and find out that the index
> > > "searching"
> > > > is significantly (4-5 times) slower in the latest version.
> > > > It seems that
> > > > org.apache.lucene.search.IndexSearcher#doc(int)
> > > > is slower.
> > > >
> > > > Is it possible to have similar performance with 8.7.0?
> > > >
> > > > Best regards,
> > > > Martynas
> > > >
> > >
> > >
> > > --
> > > Adrien
> > >
> >
>
>
> --
> Vincenzo D'Amore
>

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Martynas L <ma...@gmail.com>.
Query is fast, but document retrieval is "slow".
We call:
1) IndexSearcher#search(Query, Collector) to collect docIDs, and then
2) retrieve documents with IndexSearcher#doc(int).

In our case (1) takes less than 0.5s, while (2) almost 1.5 min (4 times
slower than 7.5.0)

On Tue, Dec 22, 2020 at 3:23 PM Vincenzo D'Amore <v....@gmail.com> wrote:

> I think it would be useful to have an example of a document and, if
> possible, an example of query that takes too long.
>
> On Mon, Dec 21, 2020 at 1:47 PM Martynas L <ma...@gmail.com> wrote:
>
> > Hello,
> >
> > I am sorry for the delay.
> >
> > Not sure what you mean by "workload". We have a performance tests, which
> > started failing after upgrading to 8.7.0.
> > So I just tried to query the index (built form the same source) to get
> all
> > documents and compare the performance with 7.5.0.
> >
> > Document "size" is a sum of all stored string lengths (3402519
> documents):
> >
> > doc size 903 - 88s vs 22s
> >
> > doc size 36 (only one field loaded, used searcher.doc(docID,
> > Collections.singleton("fieldName"))) - 78s vs 16s
> >
> > doc size 439 (some fields made not stored) - 46s vs 14.5s
> >
> > Best regards,
> > Martynas
> >
> > On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jp...@gmail.com> wrote:
> >
> > > Hello Martynas,
> > >
> > > There have indeed been changes related to stored fields in 8.7. What
> does
> > > your workload look like and how large are your documents on average?
> > >
> > > On Thu, Dec 3, 2020 at 3:04 PM Martynas L <ma...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > > We've migrated from 7.5.0 to 8.7.0 and find out that the index
> > > "searching"
> > > > is significantly (4-5 times) slower in the latest version.
> > > > It seems that
> > > > org.apache.lucene.search.IndexSearcher#doc(int)
> > > > is slower.
> > > >
> > > > Is it possible to have similar performance with 8.7.0?
> > > >
> > > > Best regards,
> > > > Martynas
> > > >
> > >
> > >
> > > --
> > > Adrien
> > >
> >
>
>
> --
> Vincenzo D'Amore
>

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Vincenzo D'Amore <v....@gmail.com>.
I think it would be useful to have an example of a document and, if
possible, an example of query that takes too long.

On Mon, Dec 21, 2020 at 1:47 PM Martynas L <ma...@gmail.com> wrote:

> Hello,
>
> I am sorry for the delay.
>
> Not sure what you mean by "workload". We have a performance tests, which
> started failing after upgrading to 8.7.0.
> So I just tried to query the index (built form the same source) to get all
> documents and compare the performance with 7.5.0.
>
> Document "size" is a sum of all stored string lengths (3402519 documents):
>
> doc size 903 - 88s vs 22s
>
> doc size 36 (only one field loaded, used searcher.doc(docID,
> Collections.singleton("fieldName"))) - 78s vs 16s
>
> doc size 439 (some fields made not stored) - 46s vs 14.5s
>
> Best regards,
> Martynas
>
> On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jp...@gmail.com> wrote:
>
> > Hello Martynas,
> >
> > There have indeed been changes related to stored fields in 8.7. What does
> > your workload look like and how large are your documents on average?
> >
> > On Thu, Dec 3, 2020 at 3:04 PM Martynas L <ma...@gmail.com>
> wrote:
> >
> > > Hi,
> > > We've migrated from 7.5.0 to 8.7.0 and find out that the index
> > "searching"
> > > is significantly (4-5 times) slower in the latest version.
> > > It seems that
> > > org.apache.lucene.search.IndexSearcher#doc(int)
> > > is slower.
> > >
> > > Is it possible to have similar performance with 8.7.0?
> > >
> > > Best regards,
> > > Martynas
> > >
> >
> >
> > --
> > Adrien
> >
>


-- 
Vincenzo D'Amore

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Martynas L <ma...@gmail.com>.
Hello,

I am sorry for the delay.

Not sure what you mean by "workload". We have a performance tests, which
started failing after upgrading to 8.7.0.
So I just tried to query the index (built form the same source) to get all
documents and compare the performance with 7.5.0.

Document "size" is a sum of all stored string lengths (3402519 documents):

doc size 903 - 88s vs 22s

doc size 36 (only one field loaded, used searcher.doc(docID,
Collections.singleton("fieldName"))) - 78s vs 16s

doc size 439 (some fields made not stored) - 46s vs 14.5s

Best regards,
Martynas

On Fri, Dec 4, 2020 at 12:06 AM Adrien Grand <jp...@gmail.com> wrote:

> Hello Martynas,
>
> There have indeed been changes related to stored fields in 8.7. What does
> your workload look like and how large are your documents on average?
>
> On Thu, Dec 3, 2020 at 3:04 PM Martynas L <ma...@gmail.com> wrote:
>
> > Hi,
> > We've migrated from 7.5.0 to 8.7.0 and find out that the index
> "searching"
> > is significantly (4-5 times) slower in the latest version.
> > It seems that
> > org.apache.lucene.search.IndexSearcher#doc(int)
> > is slower.
> >
> > Is it possible to have similar performance with 8.7.0?
> >
> > Best regards,
> > Martynas
> >
>
>
> --
> Adrien
>

Re: Slower document retrieval in 8.7.0 comparing to 7.5.0

Posted by Adrien Grand <jp...@gmail.com>.
Hello Martynas,

There have indeed been changes related to stored fields in 8.7. What does
your workload look like and how large are your documents on average?

On Thu, Dec 3, 2020 at 3:04 PM Martynas L <ma...@gmail.com> wrote:

> Hi,
> We've migrated from 7.5.0 to 8.7.0 and find out that the index "searching"
> is significantly (4-5 times) slower in the latest version.
> It seems that
> org.apache.lucene.search.IndexSearcher#doc(int)
> is slower.
>
> Is it possible to have similar performance with 8.7.0?
>
> Best regards,
> Martynas
>


-- 
Adrien