You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Ufuk YILMAZ via users <us...@solr.apache.org> on 2026/01/23 12:49:52 UTC
Have a large string/text field but be able to use export handler too
I am a fan of the export handler so I try to have docValues in all my
fields to be able to export them, if it's an analyzed field, I just make
a companion copyField that stores string and has docValues.
Today I came across a situation that I couldn't solve. I have a field
that stores a very long string. If I simply configure it with:
`indexed="false" stored="false" docValues="true"
useDocValuesAsStored="true"`
when indexing it throws an error saying something like:
`DocValuesField "data" is too large, must be <= 32766`
then I thought of storing it as binary:
`<field name="data" type="binary" indexed="false" stored="false"
docValues="true" useDocValuesAsStored="true"/>`
I was able to index the large field this way. However, when I called the
export handler, it threw an error saying "field type must be one of
these: int, long, string, SortableText..." which doesn't include
"binary"
Then I tried making a stored string (type=string, stored=true,
docValues=false) which worked for indexing, but not for the export
handler due to lacking docValues (AI hallucinated that new Solr versions
support exporting stored fields...)
So, is there a way to have a large field but also have the capability of
using the export handler?
--ufuk
Re: Have a large string/text field but be able to use export handler too
Posted by ufuk yılmaz via users <us...@solr.apache.org>.
That’s exciting news, thank you for this PR!
There are many times I can accept a little performance hit but still be able to export a field (can’t do cursorMark in the middle of a streaming expression)
Something I wonder in my case, I can have a binary field that has docValues, index it and Solr doesn’t complain about it. I guess that means it already has docValues saved? Why does export handler complaints about the field type in that case? I’ll check the code to see if there is an explicit type check that omits binary fields.
—ufuk
—
> On Jan 24, 2026, at 2:23, Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) <lk...@bloomberg.net> wrote:
>
> I have this new PR open which might be of interest:
>
> https://github.com/apache/solr/pull/4053
>
>
> From: users@solr.apache.org At: 01/23/26 07:54:27 UTC-5:00To: users@solr.apache.org
> Cc: uyilmaz@vivaldi.net
> Subject: Re: Have a large string/text field but be able to use export handler too
>
> Use fetch to wrap the export stream and join on the id to add the large
> field without the docvalue.
>
> Mathew
>
>> On Fri, Jan 23, 2026, 7:50 AM Ufuk YILMAZ via users <us...@solr.apache.org>
>> wrote:
>>
>> I am a fan of the export handler so I try to have docValues in all my
>> fields to be able to export them, if it's an analyzed field, I just make
>> a companion copyField that stores string and has docValues.
>>
>> Today I came across a situation that I couldn't solve. I have a field
>> that stores a very long string. If I simply configure it with:
>>
>> `indexed="false" stored="false" docValues="true"
>> useDocValuesAsStored="true"`
>>
>> when indexing it throws an error saying something like:
>>
>> `DocValuesField "data" is too large, must be <= 32766`
>>
>> then I thought of storing it as binary:
>>
>> `<field name="data" type="binary" indexed="false" stored="false"
>> docValues="true" useDocValuesAsStored="true"/>`
>>
>> I was able to index the large field this way. However, when I called the
>> export handler, it threw an error saying "field type must be one of
>> these: int, long, string, SortableText..." which doesn't include
>> "binary"
>>
>> Then I tried making a stored string (type=string, stored=true,
>> docValues=false) which worked for indexing, but not for the export
>> handler due to lacking docValues (AI hallucinated that new Solr versions
>> support exporting stored fields...)
>>
>> So, is there a way to have a large field but also have the capability of
>> using the export handler?
>>
>>
>> --ufuk
>>
>
>
Re: Have a large string/text field but be able to use export handler too
Posted by "Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A)" <lk...@bloomberg.net>.
I have this new PR open which might be of interest:
https://github.com/apache/solr/pull/4053
From: users@solr.apache.org At: 01/23/26 07:54:27 UTC-5:00To: users@solr.apache.org
Cc: uyilmaz@vivaldi.net
Subject: Re: Have a large string/text field but be able to use export handler too
Use fetch to wrap the export stream and join on the id to add the large
field without the docvalue.
Mathew
On Fri, Jan 23, 2026, 7:50 AM Ufuk YILMAZ via users <us...@solr.apache.org>
wrote:
> I am a fan of the export handler so I try to have docValues in all my
> fields to be able to export them, if it's an analyzed field, I just make
> a companion copyField that stores string and has docValues.
>
> Today I came across a situation that I couldn't solve. I have a field
> that stores a very long string. If I simply configure it with:
>
> `indexed="false" stored="false" docValues="true"
> useDocValuesAsStored="true"`
>
> when indexing it throws an error saying something like:
>
> `DocValuesField "data" is too large, must be <= 32766`
>
> then I thought of storing it as binary:
>
> `<field name="data" type="binary" indexed="false" stored="false"
> docValues="true" useDocValuesAsStored="true"/>`
>
> I was able to index the large field this way. However, when I called the
> export handler, it threw an error saying "field type must be one of
> these: int, long, string, SortableText..." which doesn't include
> "binary"
>
> Then I tried making a stored string (type=string, stored=true,
> docValues=false) which worked for indexing, but not for the export
> handler due to lacking docValues (AI hallucinated that new Solr versions
> support exporting stored fields...)
>
> So, is there a way to have a large field but also have the capability of
> using the export handler?
>
>
> --ufuk
>
Re: Have a large string/text field but be able to use export handler too
Posted by t sornin <ts...@gmail.com>.
Use fetch to wrap the export stream and join on the id to add the large
field without the docvalue.
Mathew
On Fri, Jan 23, 2026, 7:50 AM Ufuk YILMAZ via users <us...@solr.apache.org>
wrote:
> I am a fan of the export handler so I try to have docValues in all my
> fields to be able to export them, if it's an analyzed field, I just make
> a companion copyField that stores string and has docValues.
>
> Today I came across a situation that I couldn't solve. I have a field
> that stores a very long string. If I simply configure it with:
>
> `indexed="false" stored="false" docValues="true"
> useDocValuesAsStored="true"`
>
> when indexing it throws an error saying something like:
>
> `DocValuesField "data" is too large, must be <= 32766`
>
> then I thought of storing it as binary:
>
> `<field name="data" type="binary" indexed="false" stored="false"
> docValues="true" useDocValuesAsStored="true"/>`
>
> I was able to index the large field this way. However, when I called the
> export handler, it threw an error saying "field type must be one of
> these: int, long, string, SortableText..." which doesn't include
> "binary"
>
> Then I tried making a stored string (type=string, stored=true,
> docValues=false) which worked for indexing, but not for the export
> handler due to lacking docValues (AI hallucinated that new Solr versions
> support exporting stored fields...)
>
> So, is there a way to have a large field but also have the capability of
> using the export handler?
>
>
> --ufuk
>