You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Nagendra Nagarajayya <nn...@transaxtions.com> on 2012/05/30 15:49:06 UTC
Contribute 4.0 NRT source
Hi!
I would like to contribute a more granular NRT implementation as to
soft-commit (enabled with the realtime tag). This does not close the
SolrIndexSearcher instances but keeps passing the NRT reader as a
parameter to the searcher maintaining the searcher paradigm of being
lean and stateless. Everything works as before if realtime is not
enabled. With realtime enabled, the reader passed to the searcher is an
NRT enabled reader allowing the most recent docs to be found. Since a
reader is passed as a parameter the need to close and create a
SolrIndexSearcher object is not needed and so the caches structures in
SolrIndexSearcher do not need to be invalidated. SolrIndexSearcher is a
heavy object, not closing/creating a new one, including the caches leads
to improved NRT performance.
Do I need to make a patch or should I open a JIRA and attach the source
to the JIRA ?
Regards,
Nagendra Nagarajayya
http://solr-ra.tgels.org
http://rankingalgorithm.tgels.org
Re: Contribute 4.0 NRT source
Posted by Nagendra Nagarajayya <nn...@transaxtions.com>.
David:
There is no commit with this approach, hence no closing of the
SolrIndexSearcher object. Regarding the cache invalidation, there are
some thoughts on selective dynamic update nothing has been implemented
yet. The desired cache is disabled for now depending on the search
needs. Regarding the UnInvertedField cache, it is invalidated only on
change, if realtime faceting is enabled.
I will open up a JIRA with a patch and will take a look at the tests.
Regards,
-NN
On 5/30/2012 7:22 AM, David Smiley (@MITRE.org) wrote:
> Nagendra,
>
> Simply open a JIRA issue and attach a patch. Lucene/Solr development
> largely centers around JIRA issues via the comments.
>
> Ideally you should provide tests as part of your patch; it certainly won't
> be committed if sufficient tests are never furnished. If your NRT
> implementation really works, then I suppose you could modify existing tests
> to do an NRT commit, then those tests should work. I am not suggesting
> modifying every test, instead I am suggesting modifying a solrconfig.xml
> used in many tests or modify some test infrastructure code to trigger a
> commit via your method vs. the standard path. Even if this isn't committed
> as such (e.g. committing all tests to do NRT), seeing existing tests pass
> will give confidence in your approach.
>
> NRT isn't my area of expertise, but I strongly suspect there are critical
> false assumptions with your approach if Yonik et. all didn't use this
> approach. For example, you speak of not needing to invalidate caches
> associated with SolrIndexSearcher on an NRT commit. But shouldn't most of
> those caches be invalid if you are committing new data you want searchable?
> An example is the UnInvertedField cache for faceting on multi-valued fields.
> Anyway; this is a conversation better done in JIRA because it is more
> traceable to your contribution.
>
> ~ David
>
> -----
> Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Contribute-4-0-NRT-source-tp3986838p3986845.html
> Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Contribute 4.0 NRT source
Posted by "David Smiley (@MITRE.org)" <DS...@mitre.org>.
Nagendra,
Simply open a JIRA issue and attach a patch. Lucene/Solr development
largely centers around JIRA issues via the comments.
Ideally you should provide tests as part of your patch; it certainly won't
be committed if sufficient tests are never furnished. If your NRT
implementation really works, then I suppose you could modify existing tests
to do an NRT commit, then those tests should work. I am not suggesting
modifying every test, instead I am suggesting modifying a solrconfig.xml
used in many tests or modify some test infrastructure code to trigger a
commit via your method vs. the standard path. Even if this isn't committed
as such (e.g. committing all tests to do NRT), seeing existing tests pass
will give confidence in your approach.
NRT isn't my area of expertise, but I strongly suspect there are critical
false assumptions with your approach if Yonik et. all didn't use this
approach. For example, you speak of not needing to invalidate caches
associated with SolrIndexSearcher on an NRT commit. But shouldn't most of
those caches be invalid if you are committing new data you want searchable?
An example is the UnInvertedField cache for faceting on multi-valued fields.
Anyway; this is a conversation better done in JIRA because it is more
traceable to your contribution.
~ David
-----
Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/Contribute-4-0-NRT-source-tp3986838p3986845.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org