You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Owens, Martin" <Ma...@merrillcorp.com> on 2007/12/14 16:20:07 UTC
Python Solr Writer
I'm having some trouble understanding how the solr writer intergrates into python, I can't find any examples so does anyone have any good examples of a python writer?
Best Regards, Martin Owens
Re: retrieve lucene "doc id"
Posted by Norberto Meijome <fr...@meijome.net>.
On Mon, 17 Dec 2007 14:43:55 -0500
"Norskog, Lance" <la...@divvio.com> wrote:
> We are using MD5 to generate our IDs. MD5s are 128 bits creating a very
> unique and very randomized number for the content. Nobody has ever
> reported two different data sets that create the same MD5.
yup, we use 2 Md5 concatenated . the first part is the MD5 of a group name,the 2nd part is related to the item in the group (the same item can be in different groups, so this 2nd part can also be repeated ) - of course, only 1 item can exist in each group, so it is always unique.
>
> We use the standard (some RFC) text representation of 32 hex characters.
> This has the advantage that F* pulls 1/16 of the total index, with a
> completely randomized distribution, F** 1/256, etc. This is very handy
> for data analysis and document extraction.
yup, and in our case, the first half of the docId could be used to get all items in a group. But your example is a good one - I haven't used it for that yet, but it's a simple and practical use of the doc id :)
cheers,
B
_________________________
{Beto|Norberto|Numard} Meijome
"I was born not knowing and have had only a little time to change that here and there."
Richard Feynman
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
RE: retrieve lucene "doc id"
Posted by "Norskog, Lance" <la...@divvio.com>.
We are using MD5 to generate our IDs. MD5s are 128 bits creating a very
unique and very randomized number for the content. Nobody has ever
reported two different data sets that create the same MD5.
We use the standard (some RFC) text representation of 32 hex characters.
This has the advantage that F* pulls 1/16 of the total index, with a
completely randomized distribution, F** 1/256, etc. This is very handy
for data analysis and document extraction.
MD5 creates 128 bits, but if your index is small enough that you are
willing to risk it, you could pick 64 bits and park them in a Java long.
-----Original Message-----
From: Ryan McKinley [mailto:ryantxu@gmail.com]
Sent: Monday, December 17, 2007 8:15 AM
To: solr-user@lucene.apache.org
Subject: Re: retrieve lucene "doc id"
Yonik Seeley wrote:
> On Dec 17, 2007 1:40 AM, Ben Incani <be...@datacomit.com.au>
wrote:
>> I have converted to using the Solr search interface and I am trying
>> to retrieve documents from a list of search results (where previously
>> I had used the doc id directly from the lucene query results) and the
>> solr id I have got currently indexed is unfortunately configured not
be unique!
>
> Ouch... I'd try to make a unique Id then!
> Or barring that, just try to make the query match exactly the docs you
> want back (don't do the 2 phase thing).
>
In 1.3-dev, you can use UUIDField to have solr generate a UUID for each
doc.
ryan
Re: retrieve lucene "doc id"
Posted by Ryan McKinley <ry...@gmail.com>.
Yonik Seeley wrote:
> On Dec 17, 2007 1:40 AM, Ben Incani <be...@datacomit.com.au> wrote:
>> I have converted to using the Solr search interface and I am trying to
>> retrieve documents from a list of search results (where previously I had
>> used the doc id directly from the lucene query results) and the solr id
>> I have got currently indexed is unfortunately configured not be unique!
>
> Ouch... I'd try to make a unique Id then!
> Or barring that, just try to make the query match exactly the docs you
> want back (don't do the 2 phase thing).
>
In 1.3-dev, you can use UUIDField to have solr generate a UUID for each doc.
ryan
Re: retrieve lucene "doc id"
Posted by Yonik Seeley <yo...@apache.org>.
On Dec 17, 2007 1:40 AM, Ben Incani <be...@datacomit.com.au> wrote:
> I have converted to using the Solr search interface and I am trying to
> retrieve documents from a list of search results (where previously I had
> used the doc id directly from the lucene query results) and the solr id
> I have got currently indexed is unfortunately configured not be unique!
Ouch... I'd try to make a unique Id then!
Or barring that, just try to make the query match exactly the docs you
want back (don't do the 2 phase thing).
> I do realise that lucene internal ids are transient, but for a read-only
> requests (that are not cached) should be ok.
But if you use 2 separate requests, an index can change versions
between when you get the list of ids and when you request the
documents for those ids. That's why this isn't safe as a general
feature.
Actually, it may be possible in the future... I had planned something
like for more internal use (distributed search). The first request
returns the index version, and then subsequent requests specify the
index version to ensure that the internal lucene ids remain unchanged.
-Yonik
RE: retrieve lucene "doc id"
Posted by Ben Incani <be...@datacomit.com.au>.
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf
> Of Yonik Seeley
> Sent: Monday, 17 December 2007 4:44 PM
> To: solr-user@lucene.apache.org
> Subject: Re: retrieve lucene "doc id"
>
> On Dec 16, 2007 11:40 PM, Ben Incani
> <be...@datacomit.com.au> wrote:
> > how do I retrieve the lucene "doc id" in a query?
>
> Currently that's not doable... if it was though, it would be
> a slightly dangerous feature since internal ids are transient.
> Can you explain a little more about what you are trying to do?
>
> -Yonik
>
Hi Yonik,
I have converted to using the Solr search interface and I am trying to
retrieve documents from a list of search results (where previously I had
used the doc id directly from the lucene query results) and the solr id
I have got currently indexed is unfortunately configured not be unique!
I do realise that lucene internal ids are transient, but for a read-only
requests (that are not cached) should be ok.
i have hacked org.apache.solr.request.XMLWriter.writeDoc to do a
writeInt("docId", docId).
**** code snippet ****
SolrServer server = new CommonsHttpSolrServer(solrURL);
Map<String,String> params = new HashMap<String,String>();
params.put("q", searchQuery);
params.put("rows", "20");
MapSolrParams solrParams = new MapSolrParams(params);
QueryResponse response = server.query(solrParams);
SolrDocumentList docs = response.getResults();
ArrayList<HashMap> hitsList = new ArrayList<HashMap>();
for (int i = 0; i < docs.getNumFound(); i++) {
HashMap<String,Object> resultMap = new HashMap<String,Object>();
SolrDocument doc = (SolrDocument)docs.get(i);
resultMap.put("id", doc.getFieldValue("docId"));
for(int j=0; j<resultFields.length; j++) {
String fieldName = (String)resultFields[j];
resultMap.put(fieldName, doc.getFieldValue(fieldName));
}
hitsList.add(resultMap);
}
**** code snippet ****
Re: retrieve lucene "doc id"
Posted by Yonik Seeley <yo...@apache.org>.
On Dec 16, 2007 11:40 PM, Ben Incani <be...@datacomit.com.au> wrote:
> how do I retrieve the lucene "doc id" in a query?
Currently that's not doable... if it was though, it would be a
slightly dangerous feature since internal ids are transient.
Can you explain a little more about what you are trying to do?
-Yonik
retrieve lucene "doc id"
Posted by Ben Incani <be...@datacomit.com.au>.
how do I retrieve the lucene "doc id" in a query?
-Ben
Re: Python Solr Writer
Posted by Wade Leftwich <wa...@okaynetwork.com>.
Owens, Martin wrote:
>
> I'm having some trouble understanding how the solr writer intergrates into
> python, I can't find any examples so does anyone have any good examples of
> a python writer?
>
> Best Regards, Martin Owens
>
>
If you're wondering how to use it in Python: say
response = eval(the_string_returned_by_solr)
- Wade
Ithaca NY
--
View this message in context: http://www.nabble.com/Python-Solr-Writer-tp14337808p14355869.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Python Solr Writer
Posted by Mike Klaas <mi...@gmail.com>.
Not sure if this helps, but note that the work is done in
PythonWriter, which is a subclass of JSONWriter. Most of the work is
done by JSONWriter due to the similarities of syntax.
-Mike
On 14-Dec-07, at 10:19 AM, Owens, Martin wrote:
> That would be a python solr client, not a solr writer using this:
>
> http://lucene.apache.org/solr/api/org/apache/solr/request/
> PythonResponseWriter.html
>
> Not sure how the hell it's supposed to work to be honest.
>
> -----Original Message-----
> From: ed.summers@gmail.com on behalf of Ed Summers
> Sent: Fri 12/14/2007 11:30 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Python Solr Writer
>
> Do you mean something like:
>
> http://svn.apache.org/repos/asf/lucene/solr/trunk/client/python/
> solr.py
>
> //Ed
>
> On Dec 14, 2007 10:20 AM, Owens, Martin
> <Ma...@merrillcorp.com> wrote:
>> I'm having some trouble understanding how the solr writer
>> intergrates into python, I can't find any examples so does anyone
>> have any good examples of a python writer?
>>
>> Best Regards, Martin Owens
>>
>
RE: Python Solr Writer
Posted by Chris Hostetter <ho...@fucit.org>.
: That would be a python solr client, not a solr writer using this:
:
: http://lucene.apache.org/solr/api/org/apache/solr/request/PythonResponseWriter.html
:
: Not sure how the hell it's supposed to work to be honest.
Can you clarify your question: Are you asking how the java code works
internally, or are you asking how to make Solr use it to generate Python
output? (just add "&wt=python" to your URLs, or change your
solrconfig.xml to explicitly declare the python <queryResponseWriter/> and
make it the default...
http://wiki.apache.org/solr/QueryResponseWriter
http://wiki.apache.org/solr/SolPython
-Hoss
RE: Python Solr Writer
Posted by "Owens, Martin" <Ma...@merrillcorp.com>.
That would be a python solr client, not a solr writer using this:
http://lucene.apache.org/solr/api/org/apache/solr/request/PythonResponseWriter.html
Not sure how the hell it's supposed to work to be honest.
-----Original Message-----
From: ed.summers@gmail.com on behalf of Ed Summers
Sent: Fri 12/14/2007 11:30 AM
To: solr-user@lucene.apache.org
Subject: Re: Python Solr Writer
Do you mean something like:
http://svn.apache.org/repos/asf/lucene/solr/trunk/client/python/solr.py
//Ed
On Dec 14, 2007 10:20 AM, Owens, Martin <Ma...@merrillcorp.com> wrote:
> I'm having some trouble understanding how the solr writer intergrates into python, I can't find any examples so does anyone have any good examples of a python writer?
>
> Best Regards, Martin Owens
>
Re: Python Solr Writer
Posted by Ed Summers <eh...@pobox.com>.
Do you mean something like:
http://svn.apache.org/repos/asf/lucene/solr/trunk/client/python/solr.py
//Ed
On Dec 14, 2007 10:20 AM, Owens, Martin <Ma...@merrillcorp.com> wrote:
> I'm having some trouble understanding how the solr writer intergrates into python, I can't find any examples so does anyone have any good examples of a python writer?
>
> Best Regards, Martin Owens
>