You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Simon Willnauer <si...@googlemail.com> on 2011/03/13 16:23:40 UTC
Lucene & Solr a one way street?
Hey folks,
I have recently tried to push some refactorings towards moving stuff
from Solr to modules land to enable users of Lucene to benefit from
the developments that have been made in Solr land during the past with
very little success. Actually, it was a really disappointing
experience whenever I tried to kick off issues towards this
direction. On LUCENE-2308 David asked a good question why FieldType is
not ported to Lucene rather than a new development.
I replied with:
{quote}
Moving stuff from Solr to Lucene involves lots of politics. It is way
easier to let Solr adopt eventually than fight your way through the
politics (this is my opinion though.)
{quote}
Yet, while the answer to his question doesn't matter in this context
but it raised some other question from Roberts side:
{quote}
Then why do we still have merged codebases?
If this is the way things are, then we should un-merge the two projects.
because as a lucene developer, i spend a lot of time trying to do my
part to fix various things in Solr... if its a one-way-street then we
need to un-merge.
{quote}
The discussions on LUCENE-2883 changed my personal reception on how
things work here quite dramatically. I lost almost all enthusiasm to
even try to push developments towards moving things out of Solr and
into modules since literally every movement in this direction starts a
lot of politics (at least this is my understanding drawn from the
rather non-technical disagreements I have seen folks mentioning on
this issue). I don't care where those politics come from but in my
opinion we need to find an agreement how we deal with "stealing"
functionality form Solr and make them available to lucene users. My
personal opinion is that any refactoring, consolidation of APIs etc.
should NOT be influenced by the fact that they have been Solr private
and "might" influence further development on solr with regards to
backwards compatibility etc.
Moving features to modules should be first priority and please correct
me if I am wrong this was one of the major reason why we merged the
code base. All users should benefit from the nice features which are
currently hidden in the solr code base. FunctionQuery is a tiny one
IMO and the frustration it caused on my side was immense. I don't even
wanna try to suggest to make replication, faceting or even the cloud
feature decoupled from Solr (I don't want to argue about the
feasibility or the amount of work this would be here! Just lemme say
one thing we can always branch and there is a large workforce out
there that is willing to work on stuff like that).
I can only agree with robert that if this is a one way street that the
merge makes no sense anymore. Refactoring will bring a lot of benefits
to both lucene AND solr. I also wanna quote Earwin here (I don't find
the issue right now - this quote is from my memory): "we should try to
decouple things rather than couple them even more tight". I moved out
of all Solr issues since then and I am not willing to do any work on
it until we find an agreement on this.
I should have raised this issue earlier I guess but I rather do it now
than never.
thanks for reading,
simon
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene & Solr a one way street?
Posted by Grant Ingersoll <gs...@apache.org>.
On Mar 13, 2011, at 11:47 AM, Robert Muir wrote:
> On Sun, Mar 13, 2011 at 11:47 AM, Grant Ingersoll <gs...@apache.org> wrote:
>> I guess the question people w/ Solr only hats on have (if there are such people), is which way is that street going? It seems like most people want to pull stuff out of Solr, but they don't seem to want to put into it. That's probably where some of the resistance comes from. If you want to modularize everything so that you can consume it outside of Solr, it usually means you don't use Solr, which sometimes comes across that you don't care if the modularization actually has a negative effect on Solr. I'm all for modularization and enabling everyone, but not at the cost of loss of performance in Solr. As tightly coupled as Solr is, it's pretty damn fast and resilient. Show me that you keep that whole and I'll be +1 on everything.
>
> Do you have any facts to back up these baseless accusations?
I apologize. I didn't attend to accuse anyone if it was read that way. If you read earlier, I actually thought the whole merge is going well and that their is some pretty good cross-fertilization going on. If I didn't properly convey it here, the accusations are actually against those who have only Solr hats on. Hint, I ain't one of them. It is a concern I've heard from people in the "don't poach Solr camp". I don't think it's the right attitude, but I do think it is worth mentioning the concern. I really see Lucene/Solr as a broad continuum of enabling technologies and really there isn't one or the other in my mind.
>
> Because I'll tell you how its "seems" to me: lucene committers are
> going well beyond whats required (fixing solr) to commit changes to
> lucene.
I totally agree. The sum of the parts is really awesome now.
>
> Take a look at the commits list, we are the ones doing Solr's dirty work:
> * Like Uwe Schindler fixing up tons of XML related bugs in Solr,
> fixing analysis.jsp and the related request handlers.
> * Like Simon Willnauer doing the necessary improvements to IndexReader
> such that SolrIndexReader need not exist, and trying to add good codec
> support to Solr so it can take advantage of flexible indexing.
Yep and he should commit those when he is ready.
I heartily agree this is great work.
>
> And I guess i didnt "put any effort into solr" when i spent a huge
> chunk of this weekend tracking down jre crashes and test bugs in a
> Solr cloud test?!
I never said you didn't. I am totally in awe of the work you are doing. I wish I had half the energy and focus of some of the people who commit on a regular basis.
>
> As far as modularization having a negative performance effect on Solr,
> how is this the case? Again do you have any concrete examples, or is
> this just more baseless accusations?
No, I don't. I just said those are the concerns. I tend to agree that they are unfounded.
>
> Do you have specific benchmarks to where solr's analysis is now
> somehow slower due to the refactoring (since this is the only
> modularization thats happened from solr)?!
> Doesn't look slower to me:
> http://www.lucidimagination.com/search/document/46a8351089a98aec/protwords_txt_support_in_stemmers#46a8351089a98aec
Dude, I think the analysis modularization is awesome. I'm about to begin porting it to OpenNLP for instance. I wish it was more decoupled so I wouldn't have to bring all of Lucene core over and could just bring the analysis. Likewise for Mahout.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene & Solr a one way street?
Posted by Robert Muir <rc...@gmail.com>.
On Sun, Mar 13, 2011 at 11:47 AM, Grant Ingersoll <gs...@apache.org> wrote:
> I guess the question people w/ Solr only hats on have (if there are such people), is which way is that street going? It seems like most people want to pull stuff out of Solr, but they don't seem to want to put into it. That's probably where some of the resistance comes from. If you want to modularize everything so that you can consume it outside of Solr, it usually means you don't use Solr, which sometimes comes across that you don't care if the modularization actually has a negative effect on Solr. I'm all for modularization and enabling everyone, but not at the cost of loss of performance in Solr. As tightly coupled as Solr is, it's pretty damn fast and resilient. Show me that you keep that whole and I'll be +1 on everything.
Do you have any facts to back up these baseless accusations?
Because I'll tell you how its "seems" to me: lucene committers are
going well beyond whats required (fixing solr) to commit changes to
lucene.
Take a look at the commits list, we are the ones doing Solr's dirty work:
* Like Uwe Schindler fixing up tons of XML related bugs in Solr,
fixing analysis.jsp and the related request handlers.
* Like Simon Willnauer doing the necessary improvements to IndexReader
such that SolrIndexReader need not exist, and trying to add good codec
support to Solr so it can take advantage of flexible indexing.
And I guess i didnt "put any effort into solr" when i spent a huge
chunk of this weekend tracking down jre crashes and test bugs in a
Solr cloud test?!
As far as modularization having a negative performance effect on Solr,
how is this the case? Again do you have any concrete examples, or is
this just more baseless accusations?
Do you have specific benchmarks to where solr's analysis is now
somehow slower due to the refactoring (since this is the only
modularization thats happened from solr)?!
Doesn't look slower to me:
http://www.lucidimagination.com/search/document/46a8351089a98aec/protwords_txt_support_in_stemmers#46a8351089a98aec
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene & Solr a one way street?
Posted by Simon Willnauer <si...@googlemail.com>.
On Sun, Mar 13, 2011 at 4:47 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Mar 13, 2011, at 10:23 AM, Simon Willnauer wrote:
>
>> Hey folks,
>>
>> I have recently tried to push some refactorings towards moving stuff
>> from Solr to modules land to enable users of Lucene to benefit from
>> the developments that have been made in Solr land during the past with
>> very little success. Actually, it was a really disappointing
>> experience whenever I tried to kick off issues towards this
>> direction. On LUCENE-2308 David asked a good question why FieldType is
>> not ported to Lucene rather than a new development.
>> I replied with:
>>
>> {quote}
>> Moving stuff from Solr to Lucene involves lots of politics. It is way
>> easier to let Solr adopt eventually than fight your way through the
>> politics (this is my opinion though.)
>> {quote}
>
> I hadn't looked at 2308, but to me, if there are well written patches, then they should be considered. More modules make a lot of sense to me, as long as everyone is kept whole and there are no performance losses. Moving FTs to Lucene seems like a lot of work for little benefit, but maybe that is just me. Seems like most Lucene users like to roll their own in that stuff or use Spring.
I agree it should be a lucene dev and we can catch up in solr land if
needed. But this is not about this issue I just used it to introduce
the actual point I tried to make.
>
>>
>> Yet, while the answer to his question doesn't matter in this context
>> but it raised some other question from Roberts side:
>>
>> {quote}
>> Then why do we still have merged codebases?
>> If this is the way things are, then we should un-merge the two projects.
>>
>> because as a lucene developer, i spend a lot of time trying to do my
>> part to fix various things in Solr... if its a one-way-street then we
>> need to un-merge.
>> {quote}
>>
>> The discussions on LUCENE-2883 changed my personal reception on how
>> things work here quite dramatically. I lost almost all enthusiasm to
>> even try to push developments towards moving things out of Solr and
>> into modules since literally every movement in this direction starts a
>> lot of politics (at least this is my understanding drawn from the
>> rather non-technical disagreements I have seen folks mentioning on
>> this issue). I don't care where those politics come from but in my
>> opinion we need to find an agreement how we deal with "stealing"
>> functionality form Solr and make them available to lucene users. My
>> personal opinion is that any refactoring, consolidation of APIs etc.
>> should NOT be influenced by the fact that they have been Solr private
>> and "might" influence further development on solr with regards to
>> backwards compatibility etc.
>>
>
> I actually thought 2883 was a pretty good discussion. The sum take away from it for me was "go for it". One person was hesitant about it. I think sometimes you need to just put up patches instead of having lengthy discussions.
I kindly disagree, for me it was a very frustrating, "oh well then
fuck it.." discussion and I have to admit I moved out of it quite
quickly. This issue is just an example for lots of discussion like
that where improvements are discussed to death. I agree we should let
patches speak first but I got frustrated before I could upload one.
This is the actual problem. So now you could say this is my personal
problem which is certainly true to some extend but it should not be
that way IMO. Let a patch come and discuss then would be the approach
that I favor.
>
>
>> Moving features to modules should be first priority and please correct
>> me if I am wrong this was one of the major reason why we merged the
>> code base.
>
> I don't think it is a first priority, but it is a benefit. I also don't think it was the majority reason for the merge. I think the majority reason was that most of the Solr committers were also Lucene committers and there was a fair amount of duplicated work and a desire to be on the same version. Modularization was/is also a benefit.
I disagree here as well. but I don't want to pull out those 500 emails
thread again :)
>
> FWIW, I think the merge for the most part has been successful in most places. We have better tested code, faster code, etc.
>
>> All users should benefit from the nice features which are
>> currently hidden in the solr code base. FunctionQuery is a tiny one
>> IMO and the frustration it caused on my side was immense. I don't even
>> wanna try to suggest to make replication, faceting or even the cloud
>> feature decoupled from Solr (I don't want to argue about the
>> feasibility or the amount of work this would be here! Just lemme say
>> one thing we can always branch and there is a large workforce out
>> there that is willing to work on stuff like that).
>>
>> I can only agree with robert that if this is a one way street that the
>> merge makes no sense anymore.
>
> I guess the question people w/ Solr only hats on have (if there are such people), is which way is that street going? It seems like most people want to pull stuff out of Solr, but they don't seem to want to put into it. That's probably where some of the resistance comes from. If you want to modularize everything so that you can consume it outside of Solr, it usually means you don't use Solr, which sometimes comes across that you don't care if the modularization actually has a negative effect on Solr. I'm all for modularization and enabling everyone, but not at the cost of loss of performance in Solr. As tightly coupled as Solr is, it's pretty damn fast and resilient. Show me that you keep that whole and I'll be +1 on everything.
Grant, I think this is simply not true. I see way more effort in
putting stuff into solr which comes from lucene than the other way
around. Robert has done a tremendous amount of work to improve solr
and all of us try very hard to align solr with the development in
lucene. I think it the way from lucene to solr is less visible due to
the changes in lucene always contain fixes in solr land if needed. You
know all the refactoring like on ReaderContext etc where drastic
changes in solr and almost every dev has those changes. This simply
not true but I just realized how this reception could have build up.
About the performance - I think we can sort that out!
>
> You also have to keep in mind that some of these things, replication for instance, rely on Solr things. Are you really going to more the HTTP protocols to just Lucene? What does that even mean? Lucene is a Java API. It doesn't assume containers, etc. Solr is the thing that delivers Lucene in a container.
I don't know but its not much more than replicating segments really on
a very high level. not an issue here just an example.
>
> To me, lower level things like faceting and function query belong as modules, but higher level container things belong in Solr. Not sure where schema lands, but likely still in Solr. Most Lucene users seem resistant to XML based configuration either b/c they want to code it themselves or already have a container to do that (Spring, etc.)
+1
>
>> Refactoring will bring a lot of benefits
>> to both lucene AND solr. I also wanna quote Earwin here (I don't find
>> the issue right now - this quote is from my memory): "we should try to
>> decouple things rather than couple them even more tight". I moved out
>> of all Solr issues since then and I am not willing to do any work on
>> it until we find an agreement on this.
>
> Well, I don't think that is really helpful, but you are of course free to do as you wish. Frankly, I think you should just put up patches and make commits. You have full commit rights just like everyone else. A well written patch that is thoughtful is worth 1000x any discussion on the theory of such a patch.
>
> At any rate, I appreciate the well-thought out email and glad you raised the issue.
thanks,
Simon
>
> -Grant
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene & Solr a one way street?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
On Mar 13, 2011, at 8:47 AM, Grant Ingersoll wrote:
>
> On Mar 13, 2011, at 10:23 AM, Simon Willnauer wrote:
>
>> Hey folks,
>>
>> I have recently tried to push some refactorings towards moving stuff
>> from Solr to modules land to enable users of Lucene to benefit from
>> the developments that have been made in Solr land during the past with
>> very little success. Actually, it was a really disappointing
>> experience whenever I tried to kick off issues towards this
>> direction. On LUCENE-2308 David asked a good question why FieldType is
>> not ported to Lucene rather than a new development.
>> I replied with:
>>
>> {quote}
>> Moving stuff from Solr to Lucene involves lots of politics. It is way
>> easier to let Solr adopt eventually than fight your way through the
>> politics (this is my opinion though.)
>> {quote}
>
>> Moving features to modules should be first priority and please correct
>> me if I am wrong this was one of the major reason why we merged the
>> code base.
>
> I don't think it is a first priority, but it is a benefit. I also don't think it was the majority reason for the merge. I think the majority reason was that most of the Solr committers were also Lucene committers and there was a fair amount of duplicated work and a desire to be on the same version. Modularization was/is also a benefit.
Actually the whole merge discussion started because Mike wanted to modularize the analysis code and create modules.
It's right there:
http://s.apache.org/th
http://s.apache.org/A2R
Whether since then it's not been first priority or not is of course debatable, but it's at least what kicked off the discussion and eventual action.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene & Solr a one way street?
Posted by "David Smiley (@MITRE.org)" <DS...@mitre.org>.
It's good to see this discussion finally happen. Some things are in Solr
(e.g. faceting, function queries, Yonik's recent join patch, ...) that
probably belong in Lucene. As someone contributing functionality to
Lucene/Solr in SOLR-2155 (Geospatial search via geohash prefix techniques),
anecdotally I find it most convenient for the code to span both Lucene and
Solr, particularly Solr on the testing side. (Of course each patch is
different but this is my experience) The testing infrastructure on the Solr
side is excellent. As I look to implement sorting it's going to be
difficult to have a non-Solr user take the code since the sorting capability
is wrapped up in Solr concepts like function queries.
~ David Smiley
-----
Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/Lucene-Solr-a-one-way-street-tp2672821p2674116.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Lucene & Solr a one way street?
Posted by Grant Ingersoll <gs...@apache.org>.
On Mar 13, 2011, at 10:23 AM, Simon Willnauer wrote:
> Hey folks,
>
> I have recently tried to push some refactorings towards moving stuff
> from Solr to modules land to enable users of Lucene to benefit from
> the developments that have been made in Solr land during the past with
> very little success. Actually, it was a really disappointing
> experience whenever I tried to kick off issues towards this
> direction. On LUCENE-2308 David asked a good question why FieldType is
> not ported to Lucene rather than a new development.
> I replied with:
>
> {quote}
> Moving stuff from Solr to Lucene involves lots of politics. It is way
> easier to let Solr adopt eventually than fight your way through the
> politics (this is my opinion though.)
> {quote}
I hadn't looked at 2308, but to me, if there are well written patches, then they should be considered. More modules make a lot of sense to me, as long as everyone is kept whole and there are no performance losses. Moving FTs to Lucene seems like a lot of work for little benefit, but maybe that is just me. Seems like most Lucene users like to roll their own in that stuff or use Spring.
>
> Yet, while the answer to his question doesn't matter in this context
> but it raised some other question from Roberts side:
>
> {quote}
> Then why do we still have merged codebases?
> If this is the way things are, then we should un-merge the two projects.
>
> because as a lucene developer, i spend a lot of time trying to do my
> part to fix various things in Solr... if its a one-way-street then we
> need to un-merge.
> {quote}
>
> The discussions on LUCENE-2883 changed my personal reception on how
> things work here quite dramatically. I lost almost all enthusiasm to
> even try to push developments towards moving things out of Solr and
> into modules since literally every movement in this direction starts a
> lot of politics (at least this is my understanding drawn from the
> rather non-technical disagreements I have seen folks mentioning on
> this issue). I don't care where those politics come from but in my
> opinion we need to find an agreement how we deal with "stealing"
> functionality form Solr and make them available to lucene users. My
> personal opinion is that any refactoring, consolidation of APIs etc.
> should NOT be influenced by the fact that they have been Solr private
> and "might" influence further development on solr with regards to
> backwards compatibility etc.
>
I actually thought 2883 was a pretty good discussion. The sum take away from it for me was "go for it". One person was hesitant about it. I think sometimes you need to just put up patches instead of having lengthy discussions.
> Moving features to modules should be first priority and please correct
> me if I am wrong this was one of the major reason why we merged the
> code base.
I don't think it is a first priority, but it is a benefit. I also don't think it was the majority reason for the merge. I think the majority reason was that most of the Solr committers were also Lucene committers and there was a fair amount of duplicated work and a desire to be on the same version. Modularization was/is also a benefit.
FWIW, I think the merge for the most part has been successful in most places. We have better tested code, faster code, etc.
> All users should benefit from the nice features which are
> currently hidden in the solr code base. FunctionQuery is a tiny one
> IMO and the frustration it caused on my side was immense. I don't even
> wanna try to suggest to make replication, faceting or even the cloud
> feature decoupled from Solr (I don't want to argue about the
> feasibility or the amount of work this would be here! Just lemme say
> one thing we can always branch and there is a large workforce out
> there that is willing to work on stuff like that).
>
> I can only agree with robert that if this is a one way street that the
> merge makes no sense anymore.
I guess the question people w/ Solr only hats on have (if there are such people), is which way is that street going? It seems like most people want to pull stuff out of Solr, but they don't seem to want to put into it. That's probably where some of the resistance comes from. If you want to modularize everything so that you can consume it outside of Solr, it usually means you don't use Solr, which sometimes comes across that you don't care if the modularization actually has a negative effect on Solr. I'm all for modularization and enabling everyone, but not at the cost of loss of performance in Solr. As tightly coupled as Solr is, it's pretty damn fast and resilient. Show me that you keep that whole and I'll be +1 on everything.
You also have to keep in mind that some of these things, replication for instance, rely on Solr things. Are you really going to more the HTTP protocols to just Lucene? What does that even mean? Lucene is a Java API. It doesn't assume containers, etc. Solr is the thing that delivers Lucene in a container.
To me, lower level things like faceting and function query belong as modules, but higher level container things belong in Solr. Not sure where schema lands, but likely still in Solr. Most Lucene users seem resistant to XML based configuration either b/c they want to code it themselves or already have a container to do that (Spring, etc.)
> Refactoring will bring a lot of benefits
> to both lucene AND solr. I also wanna quote Earwin here (I don't find
> the issue right now - this quote is from my memory): "we should try to
> decouple things rather than couple them even more tight". I moved out
> of all Solr issues since then and I am not willing to do any work on
> it until we find an agreement on this.
Well, I don't think that is really helpful, but you are of course free to do as you wish. Frankly, I think you should just put up patches and make commits. You have full commit rights just like everyone else. A well written patch that is thoughtful is worth 1000x any discussion on the theory of such a patch.
At any rate, I appreciate the well-thought out email and glad you raised the issue.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org