You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by Jason Cater <jc...@gnue.org> on 2007/04/26 20:34:47 UTC
Open to updates to solr.py? (Python client library)
Greetings all.
I've recently implemented a SOLR solution internally. We typically use
python as our language of choice, so I needed a python library to
connect to SOLR.
The one in your svn repository was a good starting point, but was very
low-level.
I've taken that code and extended it. Some differences from the
original solr.py:
* Requires python 2.3+
* Supports SSL (https://) schema
* Conforms (mostly) to PEP 8 -- the Python Style Guide
* Provides a high-level results object with implicit data type conversion
* Supports batching of update commands
Is there any interest in having this new library? Should I be posting
this to the issue tracker instead of solr-dev?
I've posted a tarball of an installable (python setup.py install)
version at:
http://www.gnuenterprise.org/~jcater/python-solr-1.2.0.tgz
I've also attached the .py file. If the attachment doesn't come thru, it
is viewable at:
http://www.gnuenterprise.org/~jcater/solr.py
Thanks,
Jason Cater
NCS, Inc.
Below is the doc string from my module:
-------------
A simple Solr client for python.
Features
--------
* Supports SOLR 1.2+
* Supports http/https and SSL client-side certificates
* Uses persistent HTTP connections by default
* Properly converts to/from SOLR data types, including datetime objects
* Supports both querying and update commands (add, delete).
* Supports batching of commands
* Requires python 2.3+
Connections
-----------
`SolrConnection` can be passed in the following parameters.
Only `url` is required,.
url -- URI pointing to the SOLR instance. Examples:
http://localhost:8080/solr
https://solr-server/solr
Your python install must be compiled with SSL support for the
https:// schemes to work. (Most pre-packaged pythons are.)
persistent -- Keep a persistent HTTP connection open.
Defaults to true.
timeout -- Timeout, in seconds, for the server to response.
By default, use the python default timeout (of none?)
NOTE: This changes the python-wide timeout.
ssl_key, ssl_cert -- If using client-side key files for
SSL authentication, these should be, respectively,
your PEM key file and certificate file
Once created, a connection object has the following public methods:
query (q, fields=None, highlight=None,
score=True, sort=None, **params)
q -- the query string.
fields -- optional list of fields to include. It can be either
a string in the format that SOLR expects ('id,f1,f2'), or
a python list/tuple of field names. Defaults to returning
all fields. ("*")
score -- boolean indicating whether "score" should be included
in the field list. Note that if you explicitly list
"score" in your fields value, then this parameter is
effectively ignored. Defaults to true.
highlight -- indicates whether highlighting should be included.
`highlight` can either be `False`, indicating "No" (the
default), `True`, incidating to highlight any fields
included in "fields", or a list of field names.
sort -- list of fields to sort by.
Any parameters available to SOLR 'select' calls can also be
passed in as named parameters (e.g., fq='...', rows=20, etc).
Many SOLR parameters are in a dotted notation (e.g.,
`hl.simple.post`). For such parameters, replace the dots with
underscores when calling this method. (e.g.,
hl_simple_post='</pre'>)
Returns a Response object
add(**params)
Add a document. Pass in all document fields as
keyword parameters:
add(id='foo', notes='bar')
You must "commit" for the addition to be saved.
This command honors begin_batch/end_batch.
add_many(lst)
Add a series of documents at once. Pass in a list of
dictionaries, where each dictionary is a mapping of document
fields:
add_many( [ {'id': 'foo1', 'notes': 'foo'},
{'id': 'foo2', 'notes': 'w00t'} ] )
You must "commit" for the addition to be saved.
This command honors begin_batch/end_batch.
delete(id)
Delete a document by id.
You must "commit" for the deletion to be saved.
This command honors begin_batch/end_batch.
delete_many(lst)
Delete a series of documents. Pass in a list of ids.
You must "commit" for the deletion to be saved.
This command honors begin_batch/end_batch.
delete_query(query)
Delete any documents returned by issuing a query.
You must "commit" for the deletion to be saved.
This command honors begin_batch/end_batch.
commit(wait_flush=True, wait_searcher=True)
Issue a commit command.
This command honors begin_batch/end_batch.
optimize(wait_flush=True, wait_searcher=True)
Issue an optimize command.
This command honors begin_batch/end_batch.
begin_batch()
Begin "batch" mode, in which all commands to be sent
to the SOLR server are queued up and sent all at once.
No update commands will be sent to the backend server
until end_batch() is called. Not that "query" commands
are not batched.
begin_batch/end_batch transactions can be nested.
The transaction will not be sent to the backend server
until as many end_batch() calls have been made as
begin_batch()s.
Batching is completely optional. Any update commands
issued outside of a begin_batch()/end_batch() pair will
be immediately processed.
end_batch(commit=False)
End a batching pair. Any pending commands are sent
to the backend server. If "True" is passed in to
end_batch, a <commit> is also sent.
raw_query(**params)
Send a query command (unprocessed by this library) to
the SOLR server. The resulting text is returned un-parsed.
raw_query(q='id:1', wt='python', indent='on')
Many SOLR parameters are in a dotted notation (e.g.,
`hl.simple.post`). For such parameters, replace the dots with
underscores when calling this method. (e.g.,
hl_simple_post='</pre'>)
Query Responses
---------------
Calls to connection.query() return a Response object.
Response objects always have the following properties:
results -- A list of matching documents. Each document will be a
dict of field values.
results.start -- An integer indicating the starting # of documents
results.numMatches -- An integer indicating the total # of matches.
header -- A dict containing any responseHeaders. Usually:
header['params'] -- dictionary of original parameters used to
create this response set.
header['QTime'] -- time spent on the query
header['status'] -- status code.
See SOLR documentation for other/typical return values.
This may be settable at the SOLR-level in your config files.
next_batch() -- If only a partial set of matches were returned
(by default, 10 documents at a time), then calling
.next_batch() will return a new Response object containing
the next set of matching documents. Returns None if no
more matches.
This works by re-issuing the same query to the backend server,
with a new 'start' value.
previous_batch() -- Same as next_batch, but return the previous
set of matches. Returns None if this is the first batch.
Response objects also support __len__ and iteration. So, the following
shortcuts work:
responses = connection.query('q=foo')
print len(responses)
for document in responses:
print document['id'], document['score']
If you pass in `highlight` to the SolrConnection.query call,
then the response object will also have a highlight property,
which will be a dictionary.
Quick examples on use:
----------------------
Example showing basic connection/transactions
>>> from solr import *
>>> c = SolrConnection('http://localhost:8983')
>>> c.add(id='500', name='python test doc', active=True)
>>> c.delete('123')
>>> c.commit()
Examples showing the search wrapper
>>> response = c.query('test', rows=20)
>>> print response.results.start
0
>>> for match in response:
... print match['id'],
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
>>> response = response.next_batch()
>>> print response.results.start
20
Add 3 documents and delete 1, but send all of them as a single transaction.
>>> c.begin_batch()
>>> c.add(id="1")
>>> c.add(id="2")
>>> c.add(id="3")
>>> c.delete(id="0")
>>> c.end_batch(True)
Enter a raw query, without processing the returned HTML contents.
>>> print c.query_raw(q='id:[* TO *]', wt='python', rows='10')
Re: Open to updates to solr.py? (Python client library)
Posted by Jason Cater <ja...@ncsmags.com>.
Mike,
Thanks for responding.
> Yes indeed--the python client definitely needs work. The changes you
> have below sound quite good; I'm interested in reviewing your patch.
> If you would like to contribute the client to the Solr project, you
> will need to open an issue here:
>
> http://issues.apache.org/jira/browse/SOLR
I will do that.
>
> That takes care of the licensing issues and keeps better track of the
> feedback you receive. I took a quite glance, and a few things popped
> out:
> - why not use the python response format?
I'm using the python response format for several one-off scripts (it's
quite handy!), but I wouldn't want to put such a thing in production.
There's too many security implications of blindly eval'ing code being
delivered from a server. Especially in a general-purpose library.
> - you can convert to/from solr date/time to a python one in a oneliner:
> datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
I was trying to support Python versions < 2.5, as not all major
distributions are delivering 2.5 yet. I could change to require Python
2.5 if that's preferred, however.
-- Jason
Re: Open to updates to solr.py? (Python client library)
Posted by Mike Klaas <mi...@gmail.com>.
On 4/26/07, Jason Cater <jc...@gnue.org> wrote:
> Greetings all.
>
> I've recently implemented a SOLR solution internally. We typically use
> python as our language of choice, so I needed a python library to
> connect to SOLR.
>
> The one in your svn repository was a good starting point, but was very
> low-level.
Yes indeed--the python client definitely needs work. The changes you
have below sound quite good; I'm interested in reviewing your patch.
If you would like to contribute the client to the Solr project, you
will need to open an issue here:
http://issues.apache.org/jira/browse/SOLR
That takes care of the licensing issues and keeps better track of the
feedback you receive. I took a quite glance, and a few things popped
out:
- why not use the python response format?
- you can convert to/from solr date/time to a python one in a oneliner:
datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
-Mike
Re: Open to updates to solr.py? (Python client library)
Posted by Jason Cater <jc...@gnue.org>.
Mike,
Thanks for responding.
> Yes indeed--the python client definitely needs work. The changes you
> have below sound quite good; I'm interested in reviewing your patch.
> If you would like to contribute the client to the Solr project, you
> will need to open an issue here:
>
> http://issues.apache.org/jira/browse/SOLR
I will do that.
>
> That takes care of the licensing issues and keeps better track of the
> feedback you receive. I took a quite glance, and a few things popped
> out:
> - why not use the python response format?
I'm using the python response format for several one-off scripts (it's
quite handy!), but I wouldn't want to put such a thing in production.
There's too many security implications of blindly eval'ing code being
delivered from a server. Especially in a general-purpose library.
> - you can convert to/from solr date/time to a python one in a oneliner:
> datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
I was trying to support Python versions < 2.5, as not all major
distributions are delivering 2.5 yet. I could change to require Python
2.5 if that's preferred, however.
-- Jason
Re: Open to updates to solr.py? (Python client library)
Posted by Ed Summers <eh...@pobox.com>.
On 4/26/07, Jason Cater <jc...@gnue.org> wrote:
> I've recently implemented a SOLR solution internally. We typically use
> python as our language of choice, so I needed a python library to
> connect to SOLR.
Nice work. This looks like a really nice improvement on the python
client that is currently available. I wonder, have you considered
bundling this up and making it available via PyPI so that it can be
installed w/ easy_install?
//Ed