You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Julian Hille <ju...@netimpact.de> on 2010/02/11 11:34:30 UTC

ExternalFileField

Hi,

were trying to implement another sortby Algorythm which is calculate outside of our solr Server.
Is there a limit for the lines in that outside file? Cause we sometimes have 1.5 million lines in some situations.
Also is this a performance killer for 1.5 million rows?
Most of the other files have rows about 1000-3000 thousand.

What does happen if im writing that file, and then solr tries to read it, is there somekind of timeout? does it display nothing? Or does it wait until i finish? Cause sometimes the writing could take some time (especially in the cause
when 1.5 million lines have to be written).

Last question to the ExternalFileField is, what does happen, if i start writing the externalFile while lucene trys to
sort by fields? Any exception or is there a softcache? 



Thanks for your help,
Julian Hille


---------------------------------------------------------------
NetImpact KG
Altonaer Straße 8
20357 Hamburg

Tel: 040 / 6738363 2
Mail: julian@netimpact.de

Geschäftsführer: Tarek Müller


Re: ExternalFileField

Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
>> Most of the other files have rows about 1000-3000 thousand.
>>
>> What does happen if im writing that file, and then solr tries to read it, is there somekind of timeout?
>>     
>
> Again, if I recall correctly, it loads up the file at startup or after commit (the later depends on the location of the file, either in the data dir or in the conf directory)
>
>   
In this regard, Solr 1.4 and current trunk don't work correctly when 
commit submitted.
See:

https://issues.apache.org/jira/browse/SOLR-1607

Koji

-- 
http://www.rondhuit.com/en/


Re: ExternalFileField

Posted by Grant Ingersoll <gs...@apache.org>.
On Feb 11, 2010, at 5:34 AM, Julian Hille wrote:

> Hi,
> 
> were trying to implement another sortby Algorythm which is calculate outside of our solr Server.
> Is there a limit for the lines in that outside file? Cause we sometimes have 1.5 million lines in some situations.
> Also is this a performance killer for 1.5 million rows?

I don't think it will be, but best to try.  If I recall correctly, it caches the items in memory in a similar way to the term dictionary

> Most of the other files have rows about 1000-3000 thousand.
> 
> What does happen if im writing that file, and then solr tries to read it, is there somekind of timeout?

Again, if I recall correctly, it loads up the file at startup or after commit (the later depends on the location of the file, either in the data dir or in the conf directory)

> does it display nothing? Or does it wait until i finish? Cause sometimes the writing could take some time (especially in the cause
> when 1.5 million lines have to be written).
> 
> Last question to the ExternalFileField is, what does happen, if i start writing the externalFile while lucene trys to
> sort by fields? Any exception or is there a softcache? 

Again, it is loaded during warmup, so I don't think this will be an issue.

See http://www.lucidimagination.com/search/p:solr/page:2?q=ExternalFileField for more discussions, etc.