You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Owens, Martin" <Ma...@merrillcorp.com> on 2007/12/14 16:20:07 UTC

Python Solr Writer

I'm having some trouble understanding how the solr writer intergrates into python, I can't find any examples so does anyone have any good examples of a python writer?

Best Regards, Martin Owens

Re: retrieve lucene "doc id"

Posted by Norberto Meijome <fr...@meijome.net>.
On Mon, 17 Dec 2007 14:43:55 -0500
"Norskog, Lance" <la...@divvio.com> wrote:

> We are using MD5 to generate our IDs. MD5s are 128 bits creating a very
> unique and very randomized number for the content. Nobody has ever
> reported two different data sets that create the same MD5.

yup, we use 2 Md5 concatenated . the first part is the MD5 of a group name,the 2nd part is related to the item in the group (the same item can be in different groups, so this 2nd part can also be repeated ) - of course, only 1 item can exist in each group, so it is always unique.

> 
> We use the standard (some RFC) text representation of 32 hex characters.
> This has the advantage that F* pulls 1/16 of the total index, with a
> completely randomized distribution, F**  1/256, etc.  This is very handy
> for data analysis and document extraction. 

yup, and in our case, the first half of the docId could be used to get all items in a group. But your example is a good one - I haven't used it for that yet, but it's a simple and practical  use of the doc id :)

cheers,
B
_________________________
{Beto|Norberto|Numard} Meijome

"I was born not knowing and have had only a little time to change that here and there." 
  Richard Feynman

I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.

RE: retrieve lucene "doc id"

Posted by "Norskog, Lance" <la...@divvio.com>.
We are using MD5 to generate our IDs. MD5s are 128 bits creating a very
unique and very randomized number for the content. Nobody has ever
reported two different data sets that create the same MD5.

We use the standard (some RFC) text representation of 32 hex characters.
This has the advantage that F* pulls 1/16 of the total index, with a
completely randomized distribution, F**  1/256, etc.  This is very handy
for data analysis and document extraction. 

MD5 creates 128 bits, but if your index is small enough that you are
willing to risk it, you could pick 64 bits and park them in a Java long.

-----Original Message-----
From: Ryan McKinley [mailto:ryantxu@gmail.com] 
Sent: Monday, December 17, 2007 8:15 AM
To: solr-user@lucene.apache.org
Subject: Re: retrieve lucene "doc id"

Yonik Seeley wrote:
> On Dec 17, 2007 1:40 AM, Ben Incani <be...@datacomit.com.au>
wrote:
>> I have converted to using the Solr search interface and I am trying 
>> to retrieve documents from a list of search results (where previously

>> I had used the doc id directly from the lucene query results) and the

>> solr id I have got currently indexed is unfortunately configured not
be unique!
> 
> Ouch... I'd try to make a unique Id then!
> Or barring that, just try to make the query match exactly the docs you

> want back (don't do the 2 phase thing).
> 

In 1.3-dev, you can use UUIDField to have solr generate a UUID for each
doc.

ryan

Re: retrieve lucene "doc id"

Posted by Ryan McKinley <ry...@gmail.com>.
Yonik Seeley wrote:
> On Dec 17, 2007 1:40 AM, Ben Incani <be...@datacomit.com.au> wrote:
>> I have converted to using the Solr search interface and I am trying to
>> retrieve documents from a list of search results (where previously I had
>> used the doc id directly from the lucene query results) and the solr id
>> I have got currently indexed is unfortunately configured not be unique!
> 
> Ouch... I'd try to make a unique Id then!
> Or barring that, just try to make the query match exactly the docs you
> want back (don't do the 2 phase thing).
> 

In 1.3-dev, you can use UUIDField to have solr generate a UUID for each doc.

ryan

Re: retrieve lucene "doc id"

Posted by Yonik Seeley <yo...@apache.org>.
On Dec 17, 2007 1:40 AM, Ben Incani <be...@datacomit.com.au> wrote:
> I have converted to using the Solr search interface and I am trying to
> retrieve documents from a list of search results (where previously I had
> used the doc id directly from the lucene query results) and the solr id
> I have got currently indexed is unfortunately configured not be unique!

Ouch... I'd try to make a unique Id then!
Or barring that, just try to make the query match exactly the docs you
want back (don't do the 2 phase thing).

> I do realise that lucene internal ids are transient, but for a read-only
> requests (that are not cached) should be ok.

But if you use 2 separate requests, an index can change versions
between when you get the list of ids and when you request the
documents for those ids.  That's why this isn't safe as a general
feature.

Actually, it may be possible in the future... I had planned something
like for more internal use (distributed search).  The first request
returns the index version, and then subsequent requests specify the
index version to ensure that the internal lucene ids remain unchanged.

-Yonik

RE: retrieve lucene "doc id"

Posted by Ben Incani <be...@datacomit.com.au>.
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf 
> Of Yonik Seeley
> Sent: Monday, 17 December 2007 4:44 PM
> To: solr-user@lucene.apache.org
> Subject: Re: retrieve lucene "doc id"
> 
> On Dec 16, 2007 11:40 PM, Ben Incani 
> <be...@datacomit.com.au> wrote:
> > how do I retrieve the lucene "doc id" in a query?
> 
> Currently that's not doable... if it was though, it would be 
> a slightly dangerous feature since internal ids are transient.
> Can you explain a little more about what you are trying to do?
> 
> -Yonik
> 

Hi Yonik,

I have converted to using the Solr search interface and I am trying to
retrieve documents from a list of search results (where previously I had
used the doc id directly from the lucene query results) and the solr id
I have got currently indexed is unfortunately configured not be unique!

I do realise that lucene internal ids are transient, but for a read-only
requests (that are not cached) should be ok.  

i have hacked org.apache.solr.request.XMLWriter.writeDoc to do a
writeInt("docId", docId).

**** code snippet ****

SolrServer server = new CommonsHttpSolrServer(solrURL);
Map<String,String> params = new HashMap<String,String>();
params.put("q", searchQuery);
params.put("rows", "20");

MapSolrParams solrParams = new MapSolrParams(params);    	
QueryResponse response = server.query(solrParams);

SolrDocumentList docs = response.getResults();
ArrayList<HashMap> hitsList = new ArrayList<HashMap>();

for (int i = 0; i < docs.getNumFound(); i++) {
	HashMap<String,Object> resultMap = new HashMap<String,Object>();
	SolrDocument doc = (SolrDocument)docs.get(i);
	resultMap.put("id", doc.getFieldValue("docId"));
	for(int j=0; j<resultFields.length; j++) {
    		String fieldName = (String)resultFields[j];
   		resultMap.put(fieldName, doc.getFieldValue(fieldName));
	}
	hitsList.add(resultMap);
}
**** code snippet ****

Re: retrieve lucene "doc id"

Posted by Yonik Seeley <yo...@apache.org>.
On Dec 16, 2007 11:40 PM, Ben Incani <be...@datacomit.com.au> wrote:
> how do I retrieve the lucene "doc id" in a query?

Currently that's not doable... if it was though, it would be a
slightly dangerous feature since internal ids are transient.
Can you explain a little more about what you are trying to do?

-Yonik

retrieve lucene "doc id"

Posted by Ben Incani <be...@datacomit.com.au>.
how do I retrieve the lucene "doc id" in a query?

-Ben

Re: Python Solr Writer

Posted by Wade Leftwich <wa...@okaynetwork.com>.

Owens, Martin wrote:
> 
> I'm having some trouble understanding how the solr writer intergrates into
> python, I can't find any examples so does anyone have any good examples of
> a python writer?
> 
> Best Regards, Martin Owens
> 
> 

If you're wondering how to use it in Python: say
response = eval(the_string_returned_by_solr)

- Wade
Ithaca NY

-- 
View this message in context: http://www.nabble.com/Python-Solr-Writer-tp14337808p14355869.html
Sent from the Solr - User mailing list archive at Nabble.com.


Re: Python Solr Writer

Posted by Mike Klaas <mi...@gmail.com>.
Not sure if this helps, but note that the work is done in  
PythonWriter, which is a subclass of JSONWriter.  Most of the work is  
done by JSONWriter due to the similarities of syntax.

-Mike

On 14-Dec-07, at 10:19 AM, Owens, Martin wrote:

> That would be a python solr client, not a solr writer using this:
>
> http://lucene.apache.org/solr/api/org/apache/solr/request/ 
> PythonResponseWriter.html
>
> Not sure how the hell it's supposed to work to be honest.
>
> -----Original Message-----
> From: ed.summers@gmail.com on behalf of Ed Summers
> Sent: Fri 12/14/2007 11:30 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Python Solr Writer
>
> Do you mean something like:
>
>   http://svn.apache.org/repos/asf/lucene/solr/trunk/client/python/ 
> solr.py
>
> //Ed
>
> On Dec 14, 2007 10:20 AM, Owens, Martin  
> <Ma...@merrillcorp.com> wrote:
>> I'm having some trouble understanding how the solr writer  
>> intergrates into python, I can't find any examples so does anyone  
>> have any good examples of a python writer?
>>
>> Best Regards, Martin Owens
>>
>


RE: Python Solr Writer

Posted by Chris Hostetter <ho...@fucit.org>.
: That would be a python solr client, not a solr writer using this:
: 
: http://lucene.apache.org/solr/api/org/apache/solr/request/PythonResponseWriter.html
: 
: Not sure how the hell it's supposed to work to be honest.

Can you clarify your question: Are you asking how the java code works 
internally, or are you asking how to make Solr use it to generate Python 
output?  (just add "&wt=python" to your URLs, or change your 
solrconfig.xml to explicitly declare the python <queryResponseWriter/> and 
make it the default...

	http://wiki.apache.org/solr/QueryResponseWriter
	http://wiki.apache.org/solr/SolPython


-Hoss


RE: Python Solr Writer

Posted by "Owens, Martin" <Ma...@merrillcorp.com>.
That would be a python solr client, not a solr writer using this:

http://lucene.apache.org/solr/api/org/apache/solr/request/PythonResponseWriter.html

Not sure how the hell it's supposed to work to be honest.

-----Original Message-----
From: ed.summers@gmail.com on behalf of Ed Summers
Sent: Fri 12/14/2007 11:30 AM
To: solr-user@lucene.apache.org
Subject: Re: Python Solr Writer
 
Do you mean something like:

  http://svn.apache.org/repos/asf/lucene/solr/trunk/client/python/solr.py

//Ed

On Dec 14, 2007 10:20 AM, Owens, Martin <Ma...@merrillcorp.com> wrote:
> I'm having some trouble understanding how the solr writer intergrates into python, I can't find any examples so does anyone have any good examples of a python writer?
>
> Best Regards, Martin Owens
>


Re: Python Solr Writer

Posted by Ed Summers <eh...@pobox.com>.
Do you mean something like:

  http://svn.apache.org/repos/asf/lucene/solr/trunk/client/python/solr.py

//Ed

On Dec 14, 2007 10:20 AM, Owens, Martin <Ma...@merrillcorp.com> wrote:
> I'm having some trouble understanding how the solr writer intergrates into python, I can't find any examples so does anyone have any good examples of a python writer?
>
> Best Regards, Martin Owens
>