You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Todd Long <lo...@gmail.com> on 2015/10/21 03:01:30 UTC
DIH Caching with Delta Import
It appears that DIH entity caching (e.g. SortedMapBackedCache) does not work
with deltas... is this simply a bug with the DIH cache support or somehow by
design?
Any ideas on a workaround for this? Ideally, I could just omit the
"cacheImpl" attribute but that leaves the query (using the default processor
in my case) without the appropriate where clause including the "cacheKey"
and "cacheLookup". Should SqlEntityProcessor be smart enough to ignore the
cache with deltas and simply append a where clause which includes the
"cacheKey" and "cacheLookup"? Or possibly just include a where clause which
includes ('${dih.request.command}' = 'full-import' or cacheKey =
cacheLookup)? I suppose those could be used to mitigate the issue but I was
hoping for possibly a better solution.
Any help would be greatly appreciated. Thank you.
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-Caching-with-Delta-Import-tp4235598.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH Caching with Delta Import
Posted by Todd Long <lo...@gmail.com>.
Erick Erickson wrote
> Have you considered using SolrJ instead of DIH? I've seen
> situations where that can make a difference for things like
> caching small tables at the start of a run, see:
>
> searchhub.org/2012/02/14/indexing-with-solrj/
Nice write-up. I think we're going to move to that eventually so we can
leverage our models instead of maintaining a separate data configuration.
Thank you for sharing the link.
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-Caching-with-Delta-Import-tp4235598p4238094.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH Caching with Delta Import
Posted by Erick Erickson <er...@gmail.com>.
Have you considered using SolrJ instead of DIH? I've seen
situations where that can make a difference for things like
caching small tables at the start of a run, see:
searchhub.org/2012/02/14/indexing-with-solrj/
Best,
Erick
On Sat, Oct 24, 2015 at 6:17 PM, Todd Long <lo...@gmail.com> wrote:
> Dyer, James-2 wrote
>> The DIH Cache feature does not work with delta import. Actually, much of
>> DIH does not work with delta import. The workaround you describe is
>> similar to the approach described here:
>> https://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport ,
>> which in my opinion is the best way to implement partial updates with DIH.
>
> Not what I was hoping to hear but at least that explains the delta import
> funkyness we were experiencing. Thank you for providing the partial updates
> implementation link.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/DIH-Caching-with-Delta-Import-tp4235598p4236384.html
> Sent from the Solr - User mailing list archive at Nabble.com.
RE: DIH Caching with Delta Import
Posted by Todd Long <lo...@gmail.com>.
Dyer, James-2 wrote
> The DIH Cache feature does not work with delta import. Actually, much of
> DIH does not work with delta import. The workaround you describe is
> similar to the approach described here:
> https://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport ,
> which in my opinion is the best way to implement partial updates with DIH.
Not what I was hoping to hear but at least that explains the delta import
funkyness we were experiencing. Thank you for providing the partial updates
implementation link.
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-Caching-with-Delta-Import-tp4235598p4236384.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: DIH Caching with Delta Import
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
The DIH Cache feature does not work with delta import. Actually, much of DIH does not work with delta import. The workaround you describe is similar to the approach described here: https://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport , which in my opinion is the best way to implement partial updates with DIH.
James Dyer
Ingram Content Group
-----Original Message-----
From: Todd Long [mailto:longtm@gmail.com]
Sent: Tuesday, October 20, 2015 8:02 PM
To: solr-user@lucene.apache.org
Subject: DIH Caching with Delta Import
It appears that DIH entity caching (e.g. SortedMapBackedCache) does not work
with deltas... is this simply a bug with the DIH cache support or somehow by
design?
Any ideas on a workaround for this? Ideally, I could just omit the
"cacheImpl" attribute but that leaves the query (using the default processor
in my case) without the appropriate where clause including the "cacheKey"
and "cacheLookup". Should SqlEntityProcessor be smart enough to ignore the
cache with deltas and simply append a where clause which includes the
"cacheKey" and "cacheLookup"? Or possibly just include a where clause which
includes ('${dih.request.command}' = 'full-import' or cacheKey =
cacheLookup)? I suppose those could be used to mitigate the issue but I was
hoping for possibly a better solution.
Any help would be greatly appreciated. Thank you.
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-Caching-with-Delta-Import-tp4235598.html
Sent from the Solr - User mailing list archive at Nabble.com.