maxFieldLength clarifications

[Tommaso Teofili <to...@gmail.com>]

Hi all,

I am using Solr 3.4 and I am indexing 10KB to 10MB size documents and I've
realized some of them hit the maxFieldLength settings in solrconfig.
I see from previous threads [1] some people solve this issue splitting the
document in smaller chunks and "rebuilding" it at query time with field
collapsing / result grouping feature.
Before diving into that I'd like to ask a few questions:

   1. is the maxFieldLength parameter deprecated?
   2. what is maxFieldLength counting? I understood it's counting tokens
   per document (not per field)
   3. what if I simply remove the maxFieldLength setting from the
   solrconfig?

>From what I see if I remove it from the solrconfig the text values are
still constrained to some bound since if I query the last term in a long
document's text I don't get a match.
My high level understanding was that maxFieldLength was deprecated in
IndexWriter Lucene API and moved to proper Analyzers/Filters [2].
Thanks in advance for any help.
Tommaso

[1] :
http://lucene.472066.n3.nabble.com/Can-Solr-handle-large-text-files-td3439504.html
[2] : http://www.mail-archive.com/java-user@lucene.apache.org/msg36865.html

maxFieldLength clarifications

[Tommaso Teofili <to...@gmail.com>]

Hi all,

I am using Solr 3.4 and I am indexing 10KB to 10MB size documents and I've
realized some of them hit the maxFieldLength settings in solrconfig.
I see from previous threads [1] some people solve this issue splitting the
document in smaller chunks and "rebuilding" it at query time with field
collapsing / result grouping feature.
Before diving into that I'd like to ask a few questions:

   1. is the maxFieldLength parameter deprecated?
   2. what is maxFieldLength counting? I understood it's counting tokens
   per document (not per field)
   3. what if I simply remove the maxFieldLength setting from the
   solrconfig?

>From what I see if I remove it from the solrconfig the text values are
still constrained to some bound since if I query the last term in a long
document's text I don't get a match.
My high level understanding was that maxFieldLength was deprecated in
IndexWriter Lucene API and moved to proper Analyzers/Filters [2].
Thanks in advance for any help.
Tommaso

[1] :
http://lucene.472066.n3.nabble.com/Can-Solr-handle-large-text-files-td3439504.html
[2] : http://www.mail-archive.com/java-user@lucene.apache.org/msg36865.html

p.s.:
I've sent this to solr-user@ but I got no reply so I'm trying to see if I
can get some help out here

Re: maxFieldLength clarifications

[Chris Hostetter <ho...@fucit.org>]

:    1. is the maxFieldLength parameter deprecated?
:    2. what is maxFieldLength counting? I understood it's counting tokens
:    per document (not per field)
:    3. what if I simply remove the maxFieldLength setting from the
:    solrconfig?

1. it has been deprecated and will not be used in Solr 4x, but still 
exists in Solr 3x

2. It should be terms per field per document, not just per document.

3) if you don't specify it in solrconfig.xml it defaults to "-1" which 
means no limit.

: From what I see if I remove it from the solrconfig the text values are
: still constrained to some bound since if I query the last term in a long
: document's text I don't get a match.

a) what version of solr are you using?
b) double check both the mainIndex and indexDefaults sections of your 
solrconfig.xml and make sure maxFieldLength isn't in either of them.

-Hoss