You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Brent Mills <bm...@uship.com> on 2011/12/05 21:26:49 UTC
Sharing dih "dictionaries"
I'm not really sure how to title this but here's what I'm trying to do.
I have a query that creates a rather large dictionary of codes that are shared across multiple fields of a base entity. I'm using the cachedsqlentityprocessor but I was curious if there was a way to join this multiple times to the base entity so I can avoid having to reload it for each column join.
Ex:
<entity name="parts" query="select name, code1, code2, code3 from parts">
<field column="name" name="name" />
<entity name="shareddictionary1" query="select code, description from partcodes" where="code=parts.code1">
<field column="description" name="code1desc" /></entity>
<entity name="shareddictionary2" query="select code, description from partcodes" where="code=parts.code2">
<field column="description" name="code1desc" /></entity>
<entity name="shareddictionary3" query="select code, description from partcodes" where="code=parts.code3">
<field column="description" name="code1desc" /></entity>
</entity>
Kind of a simplified example but in this case the dictionary query has to be run 3 times to join 3 different columns. It would be nice if I could load the data set once as an entity and specify how to join it in code without requiring a separate sql query. Any ideas?
Re: Sharing dih "dictionaries"
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
AFAIK DIH jar is separated from Solr war. Isn't there a chance to use DIH
from 4.0 in Solr 3.4?
James,
Sorry for hijacking the thread.
But, do you have a chance to review
https://issues.apache.org/jira/browse/SOLR-2947 I want to provide a patch
for fixing multi-threading in DIH. But formally speaking, this issue in
addition with https://issues.apache.org/jira/browse/SOLR-2933 blocks me.
Regards
On Wed, Dec 7, 2011 at 1:11 AM, Dyer, James <Ja...@ingrambook.com>wrote:
> Just FYI that the final piece of SOLR-2382 has not been committed, and
> instead has been spun off to SOLR-2943. So it you're using Trunk and you
> need the ability to persist a cache on disk and then read it back again
> later as an DIH entity, you'll need both SOLR-2943 and also a cache
> implementation. We're using the BDB-JE cache from SOLR-2613 in production.
> There is also one backed with a Lucene index (SOLR-2948).
>
> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
>
>
> -----Original Message-----
> From: Brent Mills [mailto:bmills@uship.com]
> Sent: Tuesday, December 06, 2011 2:43 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Sharing dih "dictionaries"
>
> You're totally correct. There's actually a link on the DIH page now which
> wasn't there when I had read it a long time ago. I'm really looking
> forward to 4.0, it's got a ton of great new features. Thanks for the
> links!!
>
> -----Original Message-----
> From: Mikhail Khludnev [mailto:mkhludnev@griddynamics.com]
> Sent: Monday, December 05, 2011 10:45 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Sharing dih "dictionaries"
>
> It looks like https://issues.apache.org/jira/browse/SOLR-2382 or even
> https://issues.apache.org/jira/browse/SOLR-2613.
> I guess by using SOLR-2382 you can specify your own SortedMapBackedCache
> subclass which is able to share your Dictionary.
>
> Regards
>
> On Tue, Dec 6, 2011 at 12:26 AM, Brent Mills <bm...@uship.com> wrote:
>
> > I'm not really sure how to title this but here's what I'm trying to do.
> >
> > I have a query that creates a rather large dictionary of codes that
> > are shared across multiple fields of a base entity. I'm using the
> > cachedsqlentityprocessor but I was curious if there was a way to join
> > this multiple times to the base entity so I can avoid having to reload
> > it for each column join.
> >
> > Ex:
> > <entity name="parts" query="select name, code1, code2, code3 from
> > parts"> <field column="name" name="name" />
> > <entity name="shareddictionary1" query="select code, description
> > from partcodes" where="code=parts.code1">
> > <field column="description" name="code1desc" /></entity>
> > <entity name="shareddictionary2" query="select code, description
> > from partcodes" where="code=parts.code2">
> > <field column="description" name="code1desc" /></entity>
> > <entity name="shareddictionary3" query="select code, description
> > from partcodes" where="code=parts.code3">
> > <field column="description" name="code1desc" /></entity>
> > </entity>
> >
> > Kind of a simplified example but in this case the dictionary query has
> > to be run 3 times to join 3 different columns. It would be nice if I
> > could load the data set once as an entity and specify how to join it
> > in code without requiring a separate sql query. Any ideas?
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Developer
> Grid Dynamics
> tel. 1-415-738-8644
> Skype: mkhludnev
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
--
Sincerely yours
Mikhail Khludnev
Developer
Grid Dynamics
tel. 1-415-738-8644
Skype: mkhludnev
<http://www.griddynamics.com>
<mk...@griddynamics.com>
RE: Sharing dih "dictionaries"
Posted by "Dyer, James" <Ja...@ingrambook.com>.
Just FYI that the final piece of SOLR-2382 has not been committed, and instead has been spun off to SOLR-2943. So it you're using Trunk and you need the ability to persist a cache on disk and then read it back again later as an DIH entity, you'll need both SOLR-2943 and also a cache implementation. We're using the BDB-JE cache from SOLR-2613 in production. There is also one backed with a Lucene index (SOLR-2948).
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Brent Mills [mailto:bmills@uship.com]
Sent: Tuesday, December 06, 2011 2:43 PM
To: solr-user@lucene.apache.org
Subject: RE: Sharing dih "dictionaries"
You're totally correct. There's actually a link on the DIH page now which wasn't there when I had read it a long time ago. I'm really looking forward to 4.0, it's got a ton of great new features. Thanks for the links!!
-----Original Message-----
From: Mikhail Khludnev [mailto:mkhludnev@griddynamics.com]
Sent: Monday, December 05, 2011 10:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Sharing dih "dictionaries"
It looks like https://issues.apache.org/jira/browse/SOLR-2382 or even https://issues.apache.org/jira/browse/SOLR-2613.
I guess by using SOLR-2382 you can specify your own SortedMapBackedCache subclass which is able to share your Dictionary.
Regards
On Tue, Dec 6, 2011 at 12:26 AM, Brent Mills <bm...@uship.com> wrote:
> I'm not really sure how to title this but here's what I'm trying to do.
>
> I have a query that creates a rather large dictionary of codes that
> are shared across multiple fields of a base entity. I'm using the
> cachedsqlentityprocessor but I was curious if there was a way to join
> this multiple times to the base entity so I can avoid having to reload
> it for each column join.
>
> Ex:
> <entity name="parts" query="select name, code1, code2, code3 from
> parts"> <field column="name" name="name" />
> <entity name="shareddictionary1" query="select code, description
> from partcodes" where="code=parts.code1">
> <field column="description" name="code1desc" /></entity>
> <entity name="shareddictionary2" query="select code, description
> from partcodes" where="code=parts.code2">
> <field column="description" name="code1desc" /></entity>
> <entity name="shareddictionary3" query="select code, description
> from partcodes" where="code=parts.code3">
> <field column="description" name="code1desc" /></entity>
> </entity>
>
> Kind of a simplified example but in this case the dictionary query has
> to be run 3 times to join 3 different columns. It would be nice if I
> could load the data set once as an entity and specify how to join it
> in code without requiring a separate sql query. Any ideas?
>
--
Sincerely yours
Mikhail Khludnev
Developer
Grid Dynamics
tel. 1-415-738-8644
Skype: mkhludnev
<http://www.griddynamics.com>
<mk...@griddynamics.com>
RE: Sharing dih "dictionaries"
Posted by Brent Mills <bm...@uship.com>.
You're totally correct. There's actually a link on the DIH page now which wasn't there when I had read it a long time ago. I'm really looking forward to 4.0, it's got a ton of great new features. Thanks for the links!!
-----Original Message-----
From: Mikhail Khludnev [mailto:mkhludnev@griddynamics.com]
Sent: Monday, December 05, 2011 10:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Sharing dih "dictionaries"
It looks like https://issues.apache.org/jira/browse/SOLR-2382 or even https://issues.apache.org/jira/browse/SOLR-2613.
I guess by using SOLR-2382 you can specify your own SortedMapBackedCache subclass which is able to share your Dictionary.
Regards
On Tue, Dec 6, 2011 at 12:26 AM, Brent Mills <bm...@uship.com> wrote:
> I'm not really sure how to title this but here's what I'm trying to do.
>
> I have a query that creates a rather large dictionary of codes that
> are shared across multiple fields of a base entity. I'm using the
> cachedsqlentityprocessor but I was curious if there was a way to join
> this multiple times to the base entity so I can avoid having to reload
> it for each column join.
>
> Ex:
> <entity name="parts" query="select name, code1, code2, code3 from
> parts"> <field column="name" name="name" />
> <entity name="shareddictionary1" query="select code, description
> from partcodes" where="code=parts.code1">
> <field column="description" name="code1desc" /></entity>
> <entity name="shareddictionary2" query="select code, description
> from partcodes" where="code=parts.code2">
> <field column="description" name="code1desc" /></entity>
> <entity name="shareddictionary3" query="select code, description
> from partcodes" where="code=parts.code3">
> <field column="description" name="code1desc" /></entity>
> </entity>
>
> Kind of a simplified example but in this case the dictionary query has
> to be run 3 times to join 3 different columns. It would be nice if I
> could load the data set once as an entity and specify how to join it
> in code without requiring a separate sql query. Any ideas?
>
--
Sincerely yours
Mikhail Khludnev
Developer
Grid Dynamics
tel. 1-415-738-8644
Skype: mkhludnev
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: Sharing dih "dictionaries"
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
It looks like https://issues.apache.org/jira/browse/SOLR-2382 or even
https://issues.apache.org/jira/browse/SOLR-2613.
I guess by using SOLR-2382 you can specify your own SortedMapBackedCache
subclass which is able to share your Dictionary.
Regards
On Tue, Dec 6, 2011 at 12:26 AM, Brent Mills <bm...@uship.com> wrote:
> I'm not really sure how to title this but here's what I'm trying to do.
>
> I have a query that creates a rather large dictionary of codes that are
> shared across multiple fields of a base entity. I'm using the
> cachedsqlentityprocessor but I was curious if there was a way to join this
> multiple times to the base entity so I can avoid having to reload it for
> each column join.
>
> Ex:
> <entity name="parts" query="select name, code1, code2, code3 from parts">
> <field column="name" name="name" />
> <entity name="shareddictionary1" query="select code, description from
> partcodes" where="code=parts.code1">
> <field column="description" name="code1desc" /></entity>
> <entity name="shareddictionary2" query="select code, description from
> partcodes" where="code=parts.code2">
> <field column="description" name="code1desc" /></entity>
> <entity name="shareddictionary3" query="select code, description from
> partcodes" where="code=parts.code3">
> <field column="description" name="code1desc" /></entity>
> </entity>
>
> Kind of a simplified example but in this case the dictionary query has to
> be run 3 times to join 3 different columns. It would be nice if I could
> load the data set once as an entity and specify how to join it in code
> without requiring a separate sql query. Any ideas?
>
--
Sincerely yours
Mikhail Khludnev
Developer
Grid Dynamics
tel. 1-415-738-8644
Skype: mkhludnev
<http://www.griddynamics.com>
<mk...@griddynamics.com>