You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kranti™ K K Parisa <kr...@gmail.com> on 2010/01/19 15:53:44 UTC
Urgent: SOLR Indexing missing tokens
Hi All,
I have a problem using SOLR indexing. I am trying to index 96 pages PDF file
(using PDFBox for extracting the file contents into String). But
surprisingly SOLR Indexing is not done for the full document. Means I can't
get all the token how ever the field contains the full text of the PDF as i
am storing the field along with indexing.
Is there any such limitations with SOLR indexing, please let me know at the
earliest.
Thanks in advance!
Best Regards,
Kranti K K Parisa
Re: Urgent: SOLR Indexing missing tokens
Posted by Kranti™ K K Parisa <kr...@gmail.com>.
Hi
I was doing the same mistake mentioned in this URL.
http://search.lucidimagination.com/search/document/30616a061f8c4bf6/solr_ignoring_maxfieldlength
maxFieldLength is there at 2 places. earlier changed at the indexDefaults
now changed at mainIndex section also.
it worked. Thanks Mark & Erick. I appreciate your help.
<http://search.lucidimagination.com/search/document/30616a061f8c4bf6/solr_ignoring_maxfieldlength>
Best Regards,
Kranti K K Parisa
2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
> Hi Erik,
>
> Yes, i deleted the index and re-indexed after increasing the value (i have
> restarted tomcat as well)
>
> but still no luck. but i was just wondering the field that i am trying to
> index has the complete document text in it as i am storing that. but not
> getting the complete terms/tokens into the index to perform the search.
>
> What would be the suggestible Analyzers, filters that I should check with?
>
> Currently I am using the following:
>
> <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.StandardFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.StandardFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> Please suggest
> Best Regards,
> Kranti K K Parisa
>
>
>
> On Tue, Jan 19, 2010 at 9:03 PM, Erick Erickson <er...@gmail.com>wrote:
>
>> Did you reindex the documents you examined? That limit
>> is applied when you index.
>>
>> Try searching the user list for maxfieldlength, this topic
>> has been discussed many times and you should find a
>> solution.
>>
>> HTH
>> Erick
>>
>> 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
>>
>> > Can anyone suggest/guide me on this.
>> >
>> > Best Regards,
>> > Kranti K K Parisa
>> >
>> >
>> >
>> > 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
>> >
>> > > Hi Mark,
>> > >
>> > > I changed the value to 1,000,000,000 to just test my luck.
>> > >
>> > > But unfortunately I am still not getting the index for all Token.
>> > >
>> > > Please suggest.
>> > >
>> > > Best Regards,
>> > > Kranti K K Parisa
>> > >
>> > >
>> > >
>> > > 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
>> > >
>> > > Hi Mark,
>> > >>
>> > >> As you see my config file contains the value as 10,000
>> > >> <maxFieldLength>10000</maxFieldLength>
>> > >>
>> > >> But when I check thru Lukeall jar file I can see the Term count
>> around
>> > >> 3,000.
>> > >>
>> > >> Please suggest.
>> > >>
>> > >> Best Regards,
>> > >> Kranti K K Parisa
>> > >>
>> > >>
>> > >>
>> > >> 2010/1/19 Mark Miller <ma...@gmail.com>
>> > >>
>> > >> It limits the number of tokens that will be indexed.
>> > >>>
>> > >>> Kranti™ K K Parisa wrote:
>> > >>> > Hi Mark,
>> > >>> >
>> > >>> > I really appreciate the quick reply.
>> > >>> >
>> > >>> > here is what I have in the config xml
>> > >>> >
>> > >>> > <ramBufferSizeMB>32</ramBufferSizeMB>
>> > >>> > <maxMergeDocs>2147483647</maxMergeDocs>
>> > >>> > * <maxFieldLength>10000</maxFieldLength>*
>> > >>> > <writeLockTimeout>1000</writeLockTimeout>
>> > >>> > <commitLockTimeout>10000</commitLockTimeout>
>> > >>> >
>> > >>> > Does this matter with Tokens?? Because the field I am using is
>> having
>> > >>> > the full content of the file ( I checked that using Lukeall jar
>> > file),
>> > >>> > how ever Tokens are not getting generated completely because of
>> which
>> > >>> > my search not working for the full content.
>> > >>> >
>> > >>> > Please suggest.
>> > >>> >
>> > >>> > Best Regards,
>> > >>> > Kranti K K Parisa
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Jan 19, 2010 at 8:27 PM, Mark Miller <
>> markrmiller@gmail.com
>> > >>> > <ma...@gmail.com>> wrote:
>> > >>> >
>> > >>> > Kranti™ K K Parisa wrote:
>> > >>> > > Hi All,
>> > >>> > >
>> > >>> > > I have a problem using SOLR indexing. I am trying to index
>> 96
>> > >>> > pages PDF file
>> > >>> > > (using PDFBox for extracting the file contents into String).
>> > But
>> > >>> > > surprisingly SOLR Indexing is not done for the full
>> document.
>> > >>> > Means I can't
>> > >>> > > get all the token how ever the field contains the full text
>> of
>> > >>> > the PDF as i
>> > >>> > > am storing the field along with indexing.
>> > >>> > >
>> > >>> > > Is there any such limitations with SOLR indexing, please let
>> me
>> > >>> > know at the
>> > >>> > > earliest.
>> > >>> > >
>> > >>> > > Thanks in advance!
>> > >>> > >
>> > >>> > > Best Regards,
>> > >>> > > Kranti K K Parisa
>> > >>> > >
>> > >>> > >
>> > >>> > Take a look at maxFieldLength in solrconfig.xml
>> > >>> >
>> > >>> > --
>> > >>> > - Mark
>> > >>> >
>> > >>> > http://www.lucidimagination.com
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>>
>> > >>>
>> > >>> --
>> > >>> - Mark
>> > >>>
>> > >>> http://www.lucidimagination.com
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>
Re: Urgent: SOLR Indexing missing tokens
Posted by Kranti™ K K Parisa <kr...@gmail.com>.
Hi Erik,
Yes, i deleted the index and re-indexed after increasing the value (i have
restarted tomcat as well)
but still no luck. but i was just wondering the field that i am trying to
index has the complete document text in it as i am storing that. but not
getting the complete terms/tokens into the index to perform the search.
What would be the suggestible Analyzers, filters that I should check with?
Currently I am using the following:
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StandardFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StandardFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Please suggest
Best Regards,
Kranti K K Parisa
On Tue, Jan 19, 2010 at 9:03 PM, Erick Erickson <er...@gmail.com>wrote:
> Did you reindex the documents you examined? That limit
> is applied when you index.
>
> Try searching the user list for maxfieldlength, this topic
> has been discussed many times and you should find a
> solution.
>
> HTH
> Erick
>
> 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
>
> > Can anyone suggest/guide me on this.
> >
> > Best Regards,
> > Kranti K K Parisa
> >
> >
> >
> > 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
> >
> > > Hi Mark,
> > >
> > > I changed the value to 1,000,000,000 to just test my luck.
> > >
> > > But unfortunately I am still not getting the index for all Token.
> > >
> > > Please suggest.
> > >
> > > Best Regards,
> > > Kranti K K Parisa
> > >
> > >
> > >
> > > 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
> > >
> > > Hi Mark,
> > >>
> > >> As you see my config file contains the value as 10,000
> > >> <maxFieldLength>10000</maxFieldLength>
> > >>
> > >> But when I check thru Lukeall jar file I can see the Term count around
> > >> 3,000.
> > >>
> > >> Please suggest.
> > >>
> > >> Best Regards,
> > >> Kranti K K Parisa
> > >>
> > >>
> > >>
> > >> 2010/1/19 Mark Miller <ma...@gmail.com>
> > >>
> > >> It limits the number of tokens that will be indexed.
> > >>>
> > >>> Kranti™ K K Parisa wrote:
> > >>> > Hi Mark,
> > >>> >
> > >>> > I really appreciate the quick reply.
> > >>> >
> > >>> > here is what I have in the config xml
> > >>> >
> > >>> > <ramBufferSizeMB>32</ramBufferSizeMB>
> > >>> > <maxMergeDocs>2147483647</maxMergeDocs>
> > >>> > * <maxFieldLength>10000</maxFieldLength>*
> > >>> > <writeLockTimeout>1000</writeLockTimeout>
> > >>> > <commitLockTimeout>10000</commitLockTimeout>
> > >>> >
> > >>> > Does this matter with Tokens?? Because the field I am using is
> having
> > >>> > the full content of the file ( I checked that using Lukeall jar
> > file),
> > >>> > how ever Tokens are not getting generated completely because of
> which
> > >>> > my search not working for the full content.
> > >>> >
> > >>> > Please suggest.
> > >>> >
> > >>> > Best Regards,
> > >>> > Kranti K K Parisa
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Tue, Jan 19, 2010 at 8:27 PM, Mark Miller <
> markrmiller@gmail.com
> > >>> > <ma...@gmail.com>> wrote:
> > >>> >
> > >>> > Kranti™ K K Parisa wrote:
> > >>> > > Hi All,
> > >>> > >
> > >>> > > I have a problem using SOLR indexing. I am trying to index 96
> > >>> > pages PDF file
> > >>> > > (using PDFBox for extracting the file contents into String).
> > But
> > >>> > > surprisingly SOLR Indexing is not done for the full document.
> > >>> > Means I can't
> > >>> > > get all the token how ever the field contains the full text
> of
> > >>> > the PDF as i
> > >>> > > am storing the field along with indexing.
> > >>> > >
> > >>> > > Is there any such limitations with SOLR indexing, please let
> me
> > >>> > know at the
> > >>> > > earliest.
> > >>> > >
> > >>> > > Thanks in advance!
> > >>> > >
> > >>> > > Best Regards,
> > >>> > > Kranti K K Parisa
> > >>> > >
> > >>> > >
> > >>> > Take a look at maxFieldLength in solrconfig.xml
> > >>> >
> > >>> > --
> > >>> > - Mark
> > >>> >
> > >>> > http://www.lucidimagination.com
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>> --
> > >>> - Mark
> > >>>
> > >>> http://www.lucidimagination.com
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >
> >
>
Re: Urgent: SOLR Indexing missing tokens
Posted by Erick Erickson <er...@gmail.com>.
Did you reindex the documents you examined? That limit
is applied when you index.
Try searching the user list for maxfieldlength, this topic
has been discussed many times and you should find a
solution.
HTH
Erick
2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
> Can anyone suggest/guide me on this.
>
> Best Regards,
> Kranti K K Parisa
>
>
>
> 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
>
> > Hi Mark,
> >
> > I changed the value to 1,000,000,000 to just test my luck.
> >
> > But unfortunately I am still not getting the index for all Token.
> >
> > Please suggest.
> >
> > Best Regards,
> > Kranti K K Parisa
> >
> >
> >
> > 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
> >
> > Hi Mark,
> >>
> >> As you see my config file contains the value as 10,000
> >> <maxFieldLength>10000</maxFieldLength>
> >>
> >> But when I check thru Lukeall jar file I can see the Term count around
> >> 3,000.
> >>
> >> Please suggest.
> >>
> >> Best Regards,
> >> Kranti K K Parisa
> >>
> >>
> >>
> >> 2010/1/19 Mark Miller <ma...@gmail.com>
> >>
> >> It limits the number of tokens that will be indexed.
> >>>
> >>> Kranti™ K K Parisa wrote:
> >>> > Hi Mark,
> >>> >
> >>> > I really appreciate the quick reply.
> >>> >
> >>> > here is what I have in the config xml
> >>> >
> >>> > <ramBufferSizeMB>32</ramBufferSizeMB>
> >>> > <maxMergeDocs>2147483647</maxMergeDocs>
> >>> > * <maxFieldLength>10000</maxFieldLength>*
> >>> > <writeLockTimeout>1000</writeLockTimeout>
> >>> > <commitLockTimeout>10000</commitLockTimeout>
> >>> >
> >>> > Does this matter with Tokens?? Because the field I am using is having
> >>> > the full content of the file ( I checked that using Lukeall jar
> file),
> >>> > how ever Tokens are not getting generated completely because of which
> >>> > my search not working for the full content.
> >>> >
> >>> > Please suggest.
> >>> >
> >>> > Best Regards,
> >>> > Kranti K K Parisa
> >>> >
> >>> >
> >>> >
> >>> > On Tue, Jan 19, 2010 at 8:27 PM, Mark Miller <markrmiller@gmail.com
> >>> > <ma...@gmail.com>> wrote:
> >>> >
> >>> > Kranti™ K K Parisa wrote:
> >>> > > Hi All,
> >>> > >
> >>> > > I have a problem using SOLR indexing. I am trying to index 96
> >>> > pages PDF file
> >>> > > (using PDFBox for extracting the file contents into String).
> But
> >>> > > surprisingly SOLR Indexing is not done for the full document.
> >>> > Means I can't
> >>> > > get all the token how ever the field contains the full text of
> >>> > the PDF as i
> >>> > > am storing the field along with indexing.
> >>> > >
> >>> > > Is there any such limitations with SOLR indexing, please let me
> >>> > know at the
> >>> > > earliest.
> >>> > >
> >>> > > Thanks in advance!
> >>> > >
> >>> > > Best Regards,
> >>> > > Kranti K K Parisa
> >>> > >
> >>> > >
> >>> > Take a look at maxFieldLength in solrconfig.xml
> >>> >
> >>> > --
> >>> > - Mark
> >>> >
> >>> > http://www.lucidimagination.com
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>> --
> >>> - Mark
> >>>
> >>> http://www.lucidimagination.com
> >>>
> >>>
> >>>
> >>>
> >>
> >
>
Re: Urgent: SOLR Indexing missing tokens
Posted by Kranti™ K K Parisa <kr...@gmail.com>.
Can anyone suggest/guide me on this.
Best Regards,
Kranti K K Parisa
2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
> Hi Mark,
>
> I changed the value to 1,000,000,000 to just test my luck.
>
> But unfortunately I am still not getting the index for all Token.
>
> Please suggest.
>
> Best Regards,
> Kranti K K Parisa
>
>
>
> 2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
>
> Hi Mark,
>>
>> As you see my config file contains the value as 10,000
>> <maxFieldLength>10000</maxFieldLength>
>>
>> But when I check thru Lukeall jar file I can see the Term count around
>> 3,000.
>>
>> Please suggest.
>>
>> Best Regards,
>> Kranti K K Parisa
>>
>>
>>
>> 2010/1/19 Mark Miller <ma...@gmail.com>
>>
>> It limits the number of tokens that will be indexed.
>>>
>>> Kranti™ K K Parisa wrote:
>>> > Hi Mark,
>>> >
>>> > I really appreciate the quick reply.
>>> >
>>> > here is what I have in the config xml
>>> >
>>> > <ramBufferSizeMB>32</ramBufferSizeMB>
>>> > <maxMergeDocs>2147483647</maxMergeDocs>
>>> > * <maxFieldLength>10000</maxFieldLength>*
>>> > <writeLockTimeout>1000</writeLockTimeout>
>>> > <commitLockTimeout>10000</commitLockTimeout>
>>> >
>>> > Does this matter with Tokens?? Because the field I am using is having
>>> > the full content of the file ( I checked that using Lukeall jar file),
>>> > how ever Tokens are not getting generated completely because of which
>>> > my search not working for the full content.
>>> >
>>> > Please suggest.
>>> >
>>> > Best Regards,
>>> > Kranti K K Parisa
>>> >
>>> >
>>> >
>>> > On Tue, Jan 19, 2010 at 8:27 PM, Mark Miller <markrmiller@gmail.com
>>> > <ma...@gmail.com>> wrote:
>>> >
>>> > Kranti™ K K Parisa wrote:
>>> > > Hi All,
>>> > >
>>> > > I have a problem using SOLR indexing. I am trying to index 96
>>> > pages PDF file
>>> > > (using PDFBox for extracting the file contents into String). But
>>> > > surprisingly SOLR Indexing is not done for the full document.
>>> > Means I can't
>>> > > get all the token how ever the field contains the full text of
>>> > the PDF as i
>>> > > am storing the field along with indexing.
>>> > >
>>> > > Is there any such limitations with SOLR indexing, please let me
>>> > know at the
>>> > > earliest.
>>> > >
>>> > > Thanks in advance!
>>> > >
>>> > > Best Regards,
>>> > > Kranti K K Parisa
>>> > >
>>> > >
>>> > Take a look at maxFieldLength in solrconfig.xml
>>> >
>>> > --
>>> > - Mark
>>> >
>>> > http://www.lucidimagination.com
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>
>
Re: Urgent: SOLR Indexing missing tokens
Posted by Kranti™ K K Parisa <kr...@gmail.com>.
Hi Mark,
I changed the value to 1,000,000,000 to just test my luck.
But unfortunately I am still not getting the index for all Token.
Please suggest.
Best Regards,
Kranti K K Parisa
2010/1/19 Kranti™ K K Parisa <kr...@gmail.com>
> Hi Mark,
>
> As you see my config file contains the value as 10,000
> <maxFieldLength>10000</maxFieldLength>
>
> But when I check thru Lukeall jar file I can see the Term count around
> 3,000.
>
> Please suggest.
>
> Best Regards,
> Kranti K K Parisa
>
>
>
> 2010/1/19 Mark Miller <ma...@gmail.com>
>
> It limits the number of tokens that will be indexed.
>>
>> Kranti™ K K Parisa wrote:
>> > Hi Mark,
>> >
>> > I really appreciate the quick reply.
>> >
>> > here is what I have in the config xml
>> >
>> > <ramBufferSizeMB>32</ramBufferSizeMB>
>> > <maxMergeDocs>2147483647</maxMergeDocs>
>> > * <maxFieldLength>10000</maxFieldLength>*
>> > <writeLockTimeout>1000</writeLockTimeout>
>> > <commitLockTimeout>10000</commitLockTimeout>
>> >
>> > Does this matter with Tokens?? Because the field I am using is having
>> > the full content of the file ( I checked that using Lukeall jar file),
>> > how ever Tokens are not getting generated completely because of which
>> > my search not working for the full content.
>> >
>> > Please suggest.
>> >
>> > Best Regards,
>> > Kranti K K Parisa
>> >
>> >
>> >
>> > On Tue, Jan 19, 2010 at 8:27 PM, Mark Miller <markrmiller@gmail.com
>> > <ma...@gmail.com>> wrote:
>> >
>> > Kranti™ K K Parisa wrote:
>> > > Hi All,
>> > >
>> > > I have a problem using SOLR indexing. I am trying to index 96
>> > pages PDF file
>> > > (using PDFBox for extracting the file contents into String). But
>> > > surprisingly SOLR Indexing is not done for the full document.
>> > Means I can't
>> > > get all the token how ever the field contains the full text of
>> > the PDF as i
>> > > am storing the field along with indexing.
>> > >
>> > > Is there any such limitations with SOLR indexing, please let me
>> > know at the
>> > > earliest.
>> > >
>> > > Thanks in advance!
>> > >
>> > > Best Regards,
>> > > Kranti K K Parisa
>> > >
>> > >
>> > Take a look at maxFieldLength in solrconfig.xml
>> >
>> > --
>> > - Mark
>> >
>> > http://www.lucidimagination.com
>> >
>> >
>> >
>> >
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>
Re: Urgent: SOLR Indexing missing tokens
Posted by Kranti™ K K Parisa <kr...@gmail.com>.
Hi Mark,
As you see my config file contains the value as 10,000
<maxFieldLength>10000</maxFieldLength>
But when I check thru Lukeall jar file I can see the Term count around
3,000.
Please suggest.
Best Regards,
Kranti K K Parisa
2010/1/19 Mark Miller <ma...@gmail.com>
> It limits the number of tokens that will be indexed.
>
> Kranti™ K K Parisa wrote:
> > Hi Mark,
> >
> > I really appreciate the quick reply.
> >
> > here is what I have in the config xml
> >
> > <ramBufferSizeMB>32</ramBufferSizeMB>
> > <maxMergeDocs>2147483647</maxMergeDocs>
> > * <maxFieldLength>10000</maxFieldLength>*
> > <writeLockTimeout>1000</writeLockTimeout>
> > <commitLockTimeout>10000</commitLockTimeout>
> >
> > Does this matter with Tokens?? Because the field I am using is having
> > the full content of the file ( I checked that using Lukeall jar file),
> > how ever Tokens are not getting generated completely because of which
> > my search not working for the full content.
> >
> > Please suggest.
> >
> > Best Regards,
> > Kranti K K Parisa
> >
> >
> >
> > On Tue, Jan 19, 2010 at 8:27 PM, Mark Miller <markrmiller@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Kranti™ K K Parisa wrote:
> > > Hi All,
> > >
> > > I have a problem using SOLR indexing. I am trying to index 96
> > pages PDF file
> > > (using PDFBox for extracting the file contents into String). But
> > > surprisingly SOLR Indexing is not done for the full document.
> > Means I can't
> > > get all the token how ever the field contains the full text of
> > the PDF as i
> > > am storing the field along with indexing.
> > >
> > > Is there any such limitations with SOLR indexing, please let me
> > know at the
> > > earliest.
> > >
> > > Thanks in advance!
> > >
> > > Best Regards,
> > > Kranti K K Parisa
> > >
> > >
> > Take a look at maxFieldLength in solrconfig.xml
> >
> > --
> > - Mark
> >
> > http://www.lucidimagination.com
> >
> >
> >
> >
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
Re: Urgent: SOLR Indexing missing tokens
Posted by Mark Miller <ma...@gmail.com>.
It limits the number of tokens that will be indexed.
Kranti™ K K Parisa wrote:
> Hi Mark,
>
> I really appreciate the quick reply.
>
> here is what I have in the config xml
>
> <ramBufferSizeMB>32</ramBufferSizeMB>
> <maxMergeDocs>2147483647</maxMergeDocs>
> * <maxFieldLength>10000</maxFieldLength>*
> <writeLockTimeout>1000</writeLockTimeout>
> <commitLockTimeout>10000</commitLockTimeout>
>
> Does this matter with Tokens?? Because the field I am using is having
> the full content of the file ( I checked that using Lukeall jar file),
> how ever Tokens are not getting generated completely because of which
> my search not working for the full content.
>
> Please suggest.
>
> Best Regards,
> Kranti K K Parisa
>
>
>
> On Tue, Jan 19, 2010 at 8:27 PM, Mark Miller <markrmiller@gmail.com
> <ma...@gmail.com>> wrote:
>
> Kranti™ K K Parisa wrote:
> > Hi All,
> >
> > I have a problem using SOLR indexing. I am trying to index 96
> pages PDF file
> > (using PDFBox for extracting the file contents into String). But
> > surprisingly SOLR Indexing is not done for the full document.
> Means I can't
> > get all the token how ever the field contains the full text of
> the PDF as i
> > am storing the field along with indexing.
> >
> > Is there any such limitations with SOLR indexing, please let me
> know at the
> > earliest.
> >
> > Thanks in advance!
> >
> > Best Regards,
> > Kranti K K Parisa
> >
> >
> Take a look at maxFieldLength in solrconfig.xml
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
--
- Mark
http://www.lucidimagination.com
Re: Urgent: SOLR Indexing missing tokens
Posted by Kranti™ K K Parisa <kr...@gmail.com>.
Hi Mark,
I really appreciate the quick reply.
here is what I have in the config xml
<ramBufferSizeMB>32</ramBufferSizeMB>
<maxMergeDocs>2147483647</maxMergeDocs>
* <maxFieldLength>10000</maxFieldLength>*
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
Does this matter with Tokens?? Because the field I am using is having the
full content of the file ( I checked that using Lukeall jar file), how ever
Tokens are not getting generated completely because of which my search not
working for the full content.
Please suggest.
Best Regards,
Kranti K K Parisa
On Tue, Jan 19, 2010 at 8:27 PM, Mark Miller <ma...@gmail.com> wrote:
> Kranti™ K K Parisa wrote:
> > Hi All,
> >
> > I have a problem using SOLR indexing. I am trying to index 96 pages PDF
> file
> > (using PDFBox for extracting the file contents into String). But
> > surprisingly SOLR Indexing is not done for the full document. Means I
> can't
> > get all the token how ever the field contains the full text of the PDF as
> i
> > am storing the field along with indexing.
> >
> > Is there any such limitations with SOLR indexing, please let me know at
> the
> > earliest.
> >
> > Thanks in advance!
> >
> > Best Regards,
> > Kranti K K Parisa
> >
> >
> Take a look at maxFieldLength in solrconfig.xml
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
Re: Urgent: SOLR Indexing missing tokens
Posted by Mark Miller <ma...@gmail.com>.
Kranti™ K K Parisa wrote:
> Hi All,
>
> I have a problem using SOLR indexing. I am trying to index 96 pages PDF file
> (using PDFBox for extracting the file contents into String). But
> surprisingly SOLR Indexing is not done for the full document. Means I can't
> get all the token how ever the field contains the full text of the PDF as i
> am storing the field along with indexing.
>
> Is there any such limitations with SOLR indexing, please let me know at the
> earliest.
>
> Thanks in advance!
>
> Best Regards,
> Kranti K K Parisa
>
>
Take a look at maxFieldLength in solrconfig.xml
--
- Mark
http://www.lucidimagination.com