You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bloodhound.apache.org by Antonia Horincar <an...@gmail.com> on 2014/03/18 00:53:19 UTC
COMDEV-108 questions
I have renamed the conversation, as I have decided to look at another project that has drawn my attention, which is COMDEV-108 (Adding Apache Solr to Bloodhound).
COMDEV-109 would have been interesting, but I’m not sure I would be able to create a thorough proposal by Friday (one that would cover all the issues that might arise when implementing a feature to make BH deployment easier).
When searching information about Solr, I came across this[1] Python client for Solr which I think would be very useful to this project. Basically it creates a Solr instance, it indexes the documents provided, and then has methods that can be used for retrieving those documents. So the project would have the following requirements:
* adding Solrpy to Bloodhound
* adding callbacks to searchable objects in Bloodhound, so that they are indexed in Apache Solr
* delegating searching to Solr (whenever someone does a search request, return the results provided by Solr, rather than the results provided by Woosh)
In my opinion, using Solrpy would be better than building the feature upon the FullTextSearchPlugin[2], as it is an open-source library with thorough documentation.
Do you think that my approach is suitable for this project?
Thanks,
Antonia
[1] https://code.google.com/p/solrpy/
[2] http://trac-hacks.org/wiki/FullTextSearchPlugin
On 17 March 2014 at 09:45:15, Ryan Ollos (ryan.ollos@wandisco.com) wrote:
On Sun, Mar 16, 2014 at 6:31 PM, Antonia Horincar <
antonia.horincar@gmail.com> wrote:
> In this case, what should be the scope of the project then? Should I stick
> on having remote attachments on Amazon S3 and Google Drive (or etc), or
> should I look further into making deployment easier?
>
I have doubts about the suitability of the remote attachments project
reflected in the ongoing discussion about whether this can be accomplished
without code changes, as well whether remote attachment storage is widely
useful itself. We can certainly come up with projects that are useful in a
limited scope, but I have doubts as to whether this is of high priority for
Bloodhound.
Making deployment easier is certainly a very high priority for the
Bloodhound project. If you are able to define a project around that
yourself, then I would say to go for it. However, I can't really give you
much guidance as to what are good approaches without doing some research
myself, and I don't think I will have time to offer much feedback or
guidance within the time frame of the submission. I would like to think
that deployment to Google App Engine is possible, but I don't know anything
about the platform, so you'll really be on your own to research that.
Certainly feel free though to send us the ideas you have, along with any
good documentation references and maybe we can offer some guidance.
So depending on how you are feeling about the projects at this state, that
potentially leaves you with not much to build on, and we certainly want you
to have a good project to run with since we've already seen that you are
very capable and follow-through on the project.
It's getting a bit late to switch directions unfortunately, but we have a
pretty good list of projects in the i.a.o issue tracker (1). If one of
those is a good fit with your interests, we can work with you to narrow the
scope and define the project over the next several days.
What Gary mentioned earlier about "remote access to subversion
repositories" is probably a good project. There is the long standing ticket
#493 (2) in Trac, and the issue of not supporting remote Subversion
repositories comes up fairly regularly on the trac-users mailing list (3).
There is also the pubsub integration (4). I can't tell you off-hand which
approach to take, but the scope is narrow enough that I could see finding
time in the next few days to offer guidance.
Anyway, I wanted to make sure you knew that even though we are short on
time, there is no shortage of projects!
Btw, you had asked about config.py. The code in there is not easy to
understand, which is a fact that I'm well aware of after trying to fix some
config.py defects in Trac over the past week. Essentially though, config.py
is a wrapper for ConfigParser (5). The classes in config.py need to be
documented better!
(1) https://issues.apache.org/bloodhound/query?status=!closed&keywords=~gsoc
(2) http://trac.edgewall.org/ticket/493
(3) https://groups.google.com/d/msg/trac-users/X-d7dZkv-QI/OmfH6ad5R14J
(4) https://issues.apache.org/bloodhound/ticket/478
(5) http://docs.python.org/2.7/library/configparser.html
Re: COMDEV-108 questions
Posted by Gary Martin <ga...@wandisco.com>.
On 17/03/14 23:53, Antonia Horincar wrote:
> I have renamed the conversation, as I have decided to look at another project that has drawn my attention, which is COMDEV-108 (Adding Apache Solr to Bloodhound).
>
> COMDEV-109 would have been interesting, but I’m not sure I would be able to create a thorough proposal by Friday (one that would cover all the issues that might arise when implementing a feature to make BH deployment easier).
>
> When searching information about Solr, I came across this[1] Python client for Solr which I think would be very useful to this project. Basically it creates a Solr instance, it indexes the documents provided, and then has methods that can be used for retrieving those documents. So the project would have the following requirements:
>
> * adding Solrpy to Bloodhound
>
> * adding callbacks to searchable objects in Bloodhound, so that they are indexed in Apache Solr
>
> * delegating searching to Solr (whenever someone does a search request, return the results provided by Solr, rather than the results provided by Woosh)
>
> In my opinion, using Solrpy would be better than building the feature upon the FullTextSearchPlugin[2], as it is an open-source library with thorough documentation.
>
> Do you think that my approach is suitable for this project?
>
> Thanks,
>
> Antonia
>
> [1] https://code.google.com/p/solrpy/
>
> [2] http://trac-hacks.org/wiki/FullTextSearchPlugin
Hi Antonia,
Yeah, I think you have the beginnings of the right approach there. This
is another of the projects that do not necessarily add any completely
new functionality but instead provides an alternative (hopefully
enhanced) search backend for Bloodhound. I would guess that using the
FullTextSearchPlugin as a base would not be suitable as there is the
existing bloodhound search functionality to integrate with. However,
that plugin does make use of the alternative
https://github.com/tow/sunburnt library for working with Solr. It looks
to me like both libraries would probably provide all the features you
need so it is either a matter of judgement and taste: are the projects
active and mature, are they easy to configure and are they easy to work
with?
I'm certainly not going to dictate the choice - either could be right in
your opinion - but you might want to take a little space in your
application to see if you can justify a choice.
The documentation for each might also be helpful:
https://pythonhosted.org/solrpy/
http://opensource.timetric.com/sunburnt/index.html
One good thing is that these libraries should reduce pretty much all of
the knowledge required of the java language but you will certainly want
to have a play with installing Solr. Given the java requirement for
installation, we would not usually expect this to be a standard install
but hopefully appropriate setups could benefit from an alternative
enhanced search.
And for quick reference for everyone else, COMDEV-108 is
https://issues.apache.org/bloodhound/ticket/488
Cheers,
Gary
Re: COMDEV-108 questions
Posted by Gary Martin <ga...@wandisco.com>.
On 10/04/14 16:25, Antonia Horincar wrote:
> Hi Gary,
>
> Thanks a lot for your response.
>
> I would like to provide a set of instructions for Solr set up for development and deployment, so that everyone would be able to use my approach for searching.
>
> What are the target platforms for deployment? Should I include Windows, along with Linux and OS X? Would be easier to write a shell script that performs the installation and setup, or just point to a set of instructions?
>
> I look forward to hearing from you.
>
> Thanks,
> Antonia
You can decide to deploy to windows if you like. If that is the case, it
might well be easier to write a python script rather than a bash script.
If there is any particular complication with this I would certainly have
no problem with you targeting Linux and OS X in the first instance. I
take it that it will be possible to run solr on a separate server so it
certainly shouldn't be the end of the world.
Instructions should be enough but it would certainly be good to get some
changes into the bloodhound setup script to add the ability to switch to
Solr or at least commands documented so that conversion can be done on
the command line.
Cheers,
Gary
Re: COMDEV-108 questions
Posted by Antonia Horincar <an...@gmail.com>.
Hi Gary,
Thanks a lot for your response.
I would like to provide a set of instructions for Solr set up for development and deployment, so that everyone would be able to use my approach for searching.
What are the target platforms for deployment? Should I include Windows, along with Linux and OS X? Would be easier to write a shell script that performs the installation and setup, or just point to a set of instructions?
I look forward to hearing from you.
Thanks,
Antonia
On 1 April 2014 at 13:33:04, Gary Martin (gary.martin@wandisco.com) wrote:
On 01/04/14 00:06, Antonia Horincar wrote:
> Hi devs,
>
> I apologise for not replying after the GSoC deadline, I had to catch up with some work in the past week. But thanks for the proposal feedback.
>
> I have a few questions regarding the project. How do you intend to install and deploy solr? Is installing and deploying it through brew install/apt-get install fine? (though I don’t know how the installation would be done on Windows). Also, what Java server would you prefer, Tomcat, or Jetty? Jetty is easier to install and configure, while Tomcat is highly customizable. I am doing an in-depth research about the technologies, and would like to hear some feedback about my ideas if possible.
>
> Thank you,
> Antonia
Hi Antonia,
I suspect that everyone would be happy with you taking the path of least
resistance with installation and deployment. As long as we can find
enough information to replicate the setup, that should be fine. It could
be good in the long run to have (or, perhaps better, be able to point
to) instructions that will show various options for installation and
deployment but that can be looked at beyond the end of the project if
necessary.
As an ASF project, Lucene are likely to provide instructions for
installing Solr from source and will probably mention one or both of
Tomcat or Jetty in it. If you prefer a package manager based
installation, that should be fine too. Unless you have a compelling
reason to go for Tomcat, if Jetty looks easier then I would go with that.
Hope that is helpful!
Cheers,
Gary
Re: COMDEV-108 questions
Posted by Gary Martin <ga...@wandisco.com>.
On 01/04/14 00:06, Antonia Horincar wrote:
> Hi devs,
>
> I apologise for not replying after the GSoC deadline, I had to catch up with some work in the past week. But thanks for the proposal feedback.
>
> I have a few questions regarding the project. How do you intend to install and deploy solr? Is installing and deploying it through brew install/apt-get install fine? (though I don’t know how the installation would be done on Windows). Also, what Java server would you prefer, Tomcat, or Jetty? Jetty is easier to install and configure, while Tomcat is highly customizable. I am doing an in-depth research about the technologies, and would like to hear some feedback about my ideas if possible.
>
> Thank you,
> Antonia
Hi Antonia,
I suspect that everyone would be happy with you taking the path of least
resistance with installation and deployment. As long as we can find
enough information to replicate the setup, that should be fine. It could
be good in the long run to have (or, perhaps better, be able to point
to) instructions that will show various options for installation and
deployment but that can be looked at beyond the end of the project if
necessary.
As an ASF project, Lucene are likely to provide instructions for
installing Solr from source and will probably mention one or both of
Tomcat or Jetty in it. If you prefer a package manager based
installation, that should be fine too. Unless you have a compelling
reason to go for Tomcat, if Jetty looks easier then I would go with that.
Hope that is helpful!
Cheers,
Gary
Re: COMDEV-108 questions
Posted by Antonia Horincar <an...@gmail.com>.
Hi devs,
I apologise for not replying after the GSoC deadline, I had to catch up with some work in the past week. But thanks for the proposal feedback.
I have a few questions regarding the project. How do you intend to install and deploy solr? Is installing and deploying it through brew install/apt-get install fine? (though I don’t know how the installation would be done on Windows). Also, what Java server would you prefer, Tomcat, or Jetty? Jetty is easier to install and configure, while Tomcat is highly customizable. I am doing an in-depth research about the technologies, and would like to hear some feedback about my ideas if possible.
Thank you,
Antonia
On 21 March 2014 at 09:54:57, Gary Martin (gary.martin@wandisco.com) wrote:
On 19/03/14 23:25, Ryan Ollos wrote:
> On Wed, Mar 19, 2014 at 4:17 PM, Antonia Horincar <
> antonia.horincar@gmail.com> wrote:
>
>> Thank you very much Gary, Anze and Ryan for your help.
>>
>> I created a first draft for my proposal, based on the information you gave
>> me and on some further research. I don't think I have the permission to
>> create a new BEP on the issue tracker (or at least I couldn't find out how
>> to create one), so I'm sharing the Google doc link so that everyone can see
>> it (
>> https://docs.google.com/document/d/1mOta9V-8Sn6Jm1-XX9z4nKojfexxFJQ-VNC8XrJZWsE/edit?usp=sharing). The proposal is not yet finalised, I still have to provide more details
>> on the implementation and also provide a timeframe for this project. I know
>> that there's not much time left until the deadline, but I would be grateful
>> if you could give me some feedback on what I've written so far.
>>
>> Thank you,
>> Antonia
>
> I'll try to take a look and get back to you within the next 24 hours. You
> can use BEP-0014 on the wiki:
> https://issues.apache.org/bloodhound/wiki/Proposals/BEP-0014
>
Antonia,
I have had a quick read of the proposal in google docs. There are some
nice bits of detail in there and I particularly like it that you have
included your own assessment of the alternative libraries already. Anze
has made a few comments on that document as well.
The ASF GSoC page (in particular this part [1]) lists the basic
expectations of an application and based on that, I think you are still
lacking an approximate schedule and information about the time that you
expect to have available or commitments that might get in the way.
The application deadline is March 21st 19:00 UTC so only about 9 hours
away now so get it submitted as quick as you can.
Cheers,
Gary
[1] http://community.apache.org/gsoc.html#applying-for-gsoc
Re: COMDEV-108 questions
Posted by Gary Martin <ga...@wandisco.com>.
On 19/03/14 23:25, Ryan Ollos wrote:
> On Wed, Mar 19, 2014 at 4:17 PM, Antonia Horincar <
> antonia.horincar@gmail.com> wrote:
>
>> Thank you very much Gary, Anze and Ryan for your help.
>>
>> I created a first draft for my proposal, based on the information you gave
>> me and on some further research. I don't think I have the permission to
>> create a new BEP on the issue tracker (or at least I couldn't find out how
>> to create one), so I'm sharing the Google doc link so that everyone can see
>> it (
>> https://docs.google.com/document/d/1mOta9V-8Sn6Jm1-XX9z4nKojfexxFJQ-VNC8XrJZWsE/edit?usp=sharing). The proposal is not yet finalised, I still have to provide more details
>> on the implementation and also provide a timeframe for this project. I know
>> that there's not much time left until the deadline, but I would be grateful
>> if you could give me some feedback on what I've written so far.
>>
>> Thank you,
>> Antonia
>
> I'll try to take a look and get back to you within the next 24 hours. You
> can use BEP-0014 on the wiki:
> https://issues.apache.org/bloodhound/wiki/Proposals/BEP-0014
>
Antonia,
I have had a quick read of the proposal in google docs. There are some
nice bits of detail in there and I particularly like it that you have
included your own assessment of the alternative libraries already. Anze
has made a few comments on that document as well.
The ASF GSoC page (in particular this part [1]) lists the basic
expectations of an application and based on that, I think you are still
lacking an approximate schedule and information about the time that you
expect to have available or commitments that might get in the way.
The application deadline is March 21st 19:00 UTC so only about 9 hours
away now so get it submitted as quick as you can.
Cheers,
Gary
[1] http://community.apache.org/gsoc.html#applying-for-gsoc
Re: COMDEV-108 questions
Posted by Ryan Ollos <ry...@wandisco.com>.
On Wed, Mar 19, 2014 at 4:17 PM, Antonia Horincar <
antonia.horincar@gmail.com> wrote:
> Thank you very much Gary, Anze and Ryan for your help.
>
> I created a first draft for my proposal, based on the information you gave
> me and on some further research. I don't think I have the permission to
> create a new BEP on the issue tracker (or at least I couldn't find out how
> to create one), so I'm sharing the Google doc link so that everyone can see
> it (
> https://docs.google.com/document/d/1mOta9V-8Sn6Jm1-XX9z4nKojfexxFJQ-VNC8XrJZWsE/edit?usp=sharing). The proposal is not yet finalised, I still have to provide more details
> on the implementation and also provide a timeframe for this project. I know
> that there's not much time left until the deadline, but I would be grateful
> if you could give me some feedback on what I've written so far.
>
> Thank you,
> Antonia
I'll try to take a look and get back to you within the next 24 hours. You
can use BEP-0014 on the wiki:
https://issues.apache.org/bloodhound/wiki/Proposals/BEP-0014
Re: COMDEV-108 questions
Posted by Antonia Horincar <an...@gmail.com>.
Thank you very much Gary, Anze and Ryan for your help.
I created a first draft for my proposal, based on the information you gave me and on some further research. I don’t think I have the permission to create a new BEP on the issue tracker (or at least I couldn’t find out how to create one), so I’m sharing the Google doc link so that everyone can see it ( https://docs.google.com/document/d/1mOta9V-8Sn6Jm1-XX9z4nKojfexxFJQ-VNC8XrJZWsE/edit?usp=sharing ). The proposal is not yet finalised, I still have to provide more details on the implementation and also provide a timeframe for this project. I know that there’s not much time left until the deadline, but I would be grateful if you could give me some feedback on what I’ve written so far.
Thank you,
Antonia
On 18 March 2014 at 20:00:28, Ryan Ollos (ryan.ollos@wandisco.com) wrote:
On Tue, Mar 18, 2014 at 6:40 AM, Anže Starič <an...@gmail.com> wrote:
> 2014-03-18 0:53 GMT+01:00 Antonia Horincar <an...@gmail.com>:
> > * adding Solrpy to Bloodhound
> >
> > * adding callbacks to searchable objects in Bloodhound, so that they are
> indexed in Apache Solr
> > * delegating searching to Solr (whenever someone does a search request,
> return the results provided by Solr, rather than the results provided by
> Woosh)
>
> Take a look at the ISearchBackend interface [1] in bhsearch. It is
> probably the easiest way to add a different search backend to
> Bloodhound.
>
> Anze
>
> [1]
> https://github.com/apache/bloodhound/blob/trunk/bloodhound_search/bhsearch/api.py#L94
>
It looks like a good project to me. I seem to remember there being quite a
bit of discussion about Solr on the mailing list in the early stages of the
BloodhoundSearchPlugin development. It might be worth a quick look over the
mailing list archives (1) to see if any of that discussion would be useful
input to your proposal. Also, in case it hasn't already been mentioned,
you'll want to read BEP-004 (2).
(1)
http://apache.markmail.org/search/?q=list%3Abloodhound+Solr#query:list%3Abloodhound%20Solr+page:1+mid:4x3yu5z2zhmlgs6e+state:results
(2) https://issues.apache.org/bloodhound/wiki/Proposals/BEP-0004
Re: COMDEV-108 questions
Posted by Ryan Ollos <ry...@wandisco.com>.
On Tue, Mar 18, 2014 at 6:40 AM, Anže Starič <an...@gmail.com> wrote:
> 2014-03-18 0:53 GMT+01:00 Antonia Horincar <an...@gmail.com>:
> > * adding Solrpy to Bloodhound
> >
> > * adding callbacks to searchable objects in Bloodhound, so that they are
> indexed in Apache Solr
> > * delegating searching to Solr (whenever someone does a search request,
> return the results provided by Solr, rather than the results provided by
> Woosh)
>
> Take a look at the ISearchBackend interface [1] in bhsearch. It is
> probably the easiest way to add a different search backend to
> Bloodhound.
>
> Anze
>
> [1]
> https://github.com/apache/bloodhound/blob/trunk/bloodhound_search/bhsearch/api.py#L94
>
It looks like a good project to me. I seem to remember there being quite a
bit of discussion about Solr on the mailing list in the early stages of the
BloodhoundSearchPlugin development. It might be worth a quick look over the
mailing list archives (1) to see if any of that discussion would be useful
input to your proposal. Also, in case it hasn't already been mentioned,
you'll want to read BEP-004 (2).
(1)
http://apache.markmail.org/search/?q=list%3Abloodhound+Solr#query:list%3Abloodhound%20Solr+page:1+mid:4x3yu5z2zhmlgs6e+state:results
(2) https://issues.apache.org/bloodhound/wiki/Proposals/BEP-0004
Re: COMDEV-108 questions
Posted by Anže Starič <an...@gmail.com>.
2014-03-18 0:53 GMT+01:00 Antonia Horincar <an...@gmail.com>:
> * adding Solrpy to Bloodhound
>
> * adding callbacks to searchable objects in Bloodhound, so that they are indexed in Apache Solr
> * delegating searching to Solr (whenever someone does a search request, return the results provided by Solr, rather than the results provided by Woosh)
Take a look at the ISearchBackend interface [1] in bhsearch. It is
probably the easiest way to add a different search backend to
Bloodhound.
Anze
[1] https://github.com/apache/bloodhound/blob/trunk/bloodhound_search/bhsearch/api.py#L94