You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Srijan <sh...@gmail.com> on 2024/01/26 10:38:31 UTC
Keeping certain stored fields uncompressed
Hi All,
I'm currently facing a significant performance challenge with Apache Solr
9.x and would greatly appreciate any insights or suggestions you might have.
Context:
In my Solr setup, I have a custom post filter that is critical to our
search process. This filter needs to read a specific stored field from each
document during the search. The field in question contains access control
information that's essential for the filter's logic.
Problem:
Since upgrading to Solr 9.x, I've observed a drastic decrease in
performance – approximately 10 to 20 times slower than before. And this
stems from the fact that stored fields in Solr 9.x are now compressed.
Decompressing these fields during each search query has introduced a
substantial performance overhead.
One potential solution for me is to use Binary Field instead of string -
since binary data is not compressed in Lucene. But the issue with this guy
is that I will have to load the entire document to read my binary field.
And since I have a large number of stored fields, this could kill my search
performance.
I'm reaching out to see if anyone in the community has encountered a
similar issue or has suggestions on how to handle this scenario? Is there a
way for me to force Lucene to keep a field uncompressed?
Thanks,
Seez
Re: Keeping certain stored fields uncompressed
Posted by Walter Underwood <wu...@wunderwood.org>.
You seem to be jumping to conclusions about causes. Might want to step back and do some measurements.
Try eliminating parts of the query one at a time, including returning fields. You might need to do this with a query set of a few thousand queries to avoid cache effects.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Jan 26, 2024, at 12:13 PM, Srijan <sh...@gmail.com> wrote:
>
> I stand corrected. Looks like my stored fields were compressed in Solr 8.11
> too. But something seems to have changed in 9.x. Decompression is awfully
> slow. New algorithm?
>
> Regarding binary field, Solr doesn't allow docvalues for binary field (btw
> Lucene does). So I tried using stored binary field but now I have to load
> the entire document to read this one stored field (and I have tons of
> stored fields)
>
>
> On Fri, Jan 26, 2024, 2:44 PM Mikhail Khludnev <mk...@apache.org> wrote:
>
>> Hello.
>> Agreed. By default it's BEST_SPEED which is LZ4. So, it can't be faster
>> or less compressive. Binary DocValues Field should be an answer.
>>
>> On Fri, Jan 26, 2024 at 9:41 PM Shawn Heisey <elyograg@elyograg.org.invalid
>>>
>> wrote:
>>
>>> On 1/26/24 03:38, Srijan wrote:
>>>> Since upgrading to Solr 9.x, I've observed a drastic decrease in
>>>> performance – approximately 10 to 20 times slower than before. And this
>>>> stems from the fact that stored fields in Solr 9.x are now compressed.
>>>> Decompressing these fields during each search query has introduced a
>>>> substantial performance overhead.
>>>
>>> Stored fields have been compressed since Solr 4.1.0. Unless you
>>> upgraded from a VERY old version, stored field compression is unlikely
>>> to be the source of your performance issue.
>>>
>>> You can have the source of the data uncompressed if you set stored to
>>> false, docValues to true, and useDocValuesAsStored to true. Not all
>>> fieldtype classes support docValues, though -- in particular TextField
>>> does not.
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>>
Re: Keeping certain stored fields uncompressed
Posted by ufuk yılmaz <uy...@vivaldi.net.INVALID>.
Hi, just curious, may I ask how did you come to the conclusion that the compression of fields is the cause of slowness in 9.4?
—
> On 26 Jan 2024, at 23:13, Srijan <sh...@gmail.com> wrote:
>
> I stand corrected. Looks like my stored fields were compressed in Solr 8.11
> too. But something seems to have changed in 9.x. Decompression is awfully
> slow. New algorithm?
>
> Regarding binary field, Solr doesn't allow docvalues for binary field (btw
> Lucene does). So I tried using stored binary field but now I have to load
> the entire document to read this one stored field (and I have tons of
> stored fields)
>
>
>> On Fri, Jan 26, 2024, 2:44 PM Mikhail Khludnev <mk...@apache.org> wrote:
>>
>> Hello.
>> Agreed. By default it's BEST_SPEED which is LZ4. So, it can't be faster
>> or less compressive. Binary DocValues Field should be an answer.
>>
>> On Fri, Jan 26, 2024 at 9:41 PM Shawn Heisey <elyograg@elyograg.org.invalid
>>>
>> wrote:
>>
>>>> On 1/26/24 03:38, Srijan wrote:
>>>> Since upgrading to Solr 9.x, I've observed a drastic decrease in
>>>> performance – approximately 10 to 20 times slower than before. And this
>>>> stems from the fact that stored fields in Solr 9.x are now compressed.
>>>> Decompressing these fields during each search query has introduced a
>>>> substantial performance overhead.
>>>
>>> Stored fields have been compressed since Solr 4.1.0. Unless you
>>> upgraded from a VERY old version, stored field compression is unlikely
>>> to be the source of your performance issue.
>>>
>>> You can have the source of the data uncompressed if you set stored to
>>> false, docValues to true, and useDocValuesAsStored to true. Not all
>>> fieldtype classes support docValues, though -- in particular TextField
>>> does not.
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>>
Re: Keeping certain stored fields uncompressed
Posted by Mikhail Khludnev <mk...@apache.org>.
On Fri, Jan 26, 2024 at 11:14 PM Srijan <sh...@gmail.com> wrote:
> Regarding binary field, Solr doesn't allow docvalues for binary field (btw
> Lucene does).
https://solr.apache.org/guide/solr/latest/indexing-guide/field-types-included-with-solr.html
mentions BinaryField
FWIW BinaryDocValues has no compression anymore
https://issues.apache.org/jira/browse/LUCENE-9843
> So I tried using stored binary field but now I have to load
> the entire document to read this one stored field (and I have tons of
> stored fields)
>
Well intentions for docValues was to read a single field, in contrast to a
stored field which always reads the whole record (document) despite the
"filter".
Presumably some Solr logic might somehow impact it, but that is what I
expect with docvalues - read a certain field only.
>
>
> On Fri, Jan 26, 2024, 2:44 PM Mikhail Khludnev <mk...@apache.org> wrote:
>
> > Hello.
> > Agreed. By default it's BEST_SPEED which is LZ4. So, it can't be faster
> > or less compressive. Binary DocValues Field should be an answer.
> >
> > On Fri, Jan 26, 2024 at 9:41 PM Shawn Heisey
> <elyograg@elyograg.org.invalid
> > >
> > wrote:
> >
> > > On 1/26/24 03:38, Srijan wrote:
> > > > Since upgrading to Solr 9.x, I've observed a drastic decrease in
> > > > performance – approximately 10 to 20 times slower than before. And
> this
> > > > stems from the fact that stored fields in Solr 9.x are now
> compressed.
> > > > Decompressing these fields during each search query has introduced a
> > > > substantial performance overhead.
> > >
> > > Stored fields have been compressed since Solr 4.1.0. Unless you
> > > upgraded from a VERY old version, stored field compression is unlikely
> > > to be the source of your performance issue.
> > >
> > > You can have the source of the data uncompressed if you set stored to
> > > false, docValues to true, and useDocValuesAsStored to true. Not all
> > > fieldtype classes support docValues, though -- in particular TextField
> > > does not.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: Keeping certain stored fields uncompressed
Posted by Srijan <sh...@gmail.com>.
I stand corrected. Looks like my stored fields were compressed in Solr 8.11
too. But something seems to have changed in 9.x. Decompression is awfully
slow. New algorithm?
Regarding binary field, Solr doesn't allow docvalues for binary field (btw
Lucene does). So I tried using stored binary field but now I have to load
the entire document to read this one stored field (and I have tons of
stored fields)
On Fri, Jan 26, 2024, 2:44 PM Mikhail Khludnev <mk...@apache.org> wrote:
> Hello.
> Agreed. By default it's BEST_SPEED which is LZ4. So, it can't be faster
> or less compressive. Binary DocValues Field should be an answer.
>
> On Fri, Jan 26, 2024 at 9:41 PM Shawn Heisey <elyograg@elyograg.org.invalid
> >
> wrote:
>
> > On 1/26/24 03:38, Srijan wrote:
> > > Since upgrading to Solr 9.x, I've observed a drastic decrease in
> > > performance – approximately 10 to 20 times slower than before. And this
> > > stems from the fact that stored fields in Solr 9.x are now compressed.
> > > Decompressing these fields during each search query has introduced a
> > > substantial performance overhead.
> >
> > Stored fields have been compressed since Solr 4.1.0. Unless you
> > upgraded from a VERY old version, stored field compression is unlikely
> > to be the source of your performance issue.
> >
> > You can have the source of the data uncompressed if you set stored to
> > false, docValues to true, and useDocValuesAsStored to true. Not all
> > fieldtype classes support docValues, though -- in particular TextField
> > does not.
> >
> > Thanks,
> > Shawn
> >
> >
>
> --
> Sincerely yours
> Mikhail Khludnev
>
Re: Keeping certain stored fields uncompressed
Posted by Mikhail Khludnev <mk...@apache.org>.
Hello.
Agreed. By default it's BEST_SPEED which is LZ4. So, it can't be faster
or less compressive. Binary DocValues Field should be an answer.
On Fri, Jan 26, 2024 at 9:41 PM Shawn Heisey <el...@elyograg.org.invalid>
wrote:
> On 1/26/24 03:38, Srijan wrote:
> > Since upgrading to Solr 9.x, I've observed a drastic decrease in
> > performance – approximately 10 to 20 times slower than before. And this
> > stems from the fact that stored fields in Solr 9.x are now compressed.
> > Decompressing these fields during each search query has introduced a
> > substantial performance overhead.
>
> Stored fields have been compressed since Solr 4.1.0. Unless you
> upgraded from a VERY old version, stored field compression is unlikely
> to be the source of your performance issue.
>
> You can have the source of the data uncompressed if you set stored to
> false, docValues to true, and useDocValuesAsStored to true. Not all
> fieldtype classes support docValues, though -- in particular TextField
> does not.
>
> Thanks,
> Shawn
>
>
--
Sincerely yours
Mikhail Khludnev
Re: Keeping certain stored fields uncompressed
Posted by Shawn Heisey <el...@elyograg.org.INVALID>.
On 1/26/24 03:38, Srijan wrote:
> Since upgrading to Solr 9.x, I've observed a drastic decrease in
> performance – approximately 10 to 20 times slower than before. And this
> stems from the fact that stored fields in Solr 9.x are now compressed.
> Decompressing these fields during each search query has introduced a
> substantial performance overhead.
Stored fields have been compressed since Solr 4.1.0. Unless you
upgraded from a VERY old version, stored field compression is unlikely
to be the source of your performance issue.
You can have the source of the data uncompressed if you set stored to
false, docValues to true, and useDocValuesAsStored to true. Not all
fieldtype classes support docValues, though -- in particular TextField
does not.
Thanks,
Shawn