You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Bennett <mb...@ideaeng.com> on 2009/09/22 19:54:31 UTC
Code sync between Lucene and Solr, crossing Apache project
boundaries, etc.
To do any serious Solr debugging (or filter development) you also need the
Solr source code tree. And you'd like them to be in sync, so that the
Lucene code you see is exactly the same as what was used for the Solr
version you're working with.
I did find this link on sync'ing the two source tress, for a specific Solr
version. But this seems a bit convoluted, is this really the right answer?
http://happygiraffe.net/blog/2009/07/16/solrs-lucene-source/
Generally:
1: Given that many threads on the Solr mailing list seem to end with "yeah,
you could write some code to do that", it seems like most developers would
face this problem, and yet I haven't seen it discussed much (at least not
with the search terms I'm using, I keep finding stuff about SolrJ or just
getting the Solr code to build)
2: Do most people pick a version of Solr, then try carefully to sync it with
the exact version of Lucene code, or do they just not worry about the
version stuff unless they get an error?
3: Or do most Solr/Lucene developers just live out of the Apache nightly
snapshots for both source code trees? And if so, I have a question about
even doing that. (see below)
4: Getting both projects to live together under Eclipse has seemed a bit
awkward, though again I'd think it's a common task. Any good links on that?
5: So do you most of you:
a: Bother with both code trees? (Solr and Lucene)
b: Live from the command line with ant? or
c: Get them both living under Eclipse? With dependencies back and forth? I
haven't found a good resource for this yet.
d: Do you also think maven, git and clover are required?
6: Since advanced developers using Nutch and Solr would typically want
Lucene's source code as well, wouldn't it be good to have separate
distribution that includes all of those? Or perhaps there's some "apache
thing" that makes this so trivial it's not worth bothering with?
Back to question 3, just using the nightly trees for both Solr and Lucene,
which presumably are in sync. If you don't need a specific release, this
might be a reasonable workaround, but still a few of the details bother me.
Get the code:
svn co http://svn.apache.org/repos/asf/lucene/java/trunk lucene-nightly
Checked out revision 817722.
svn co http://svn.apache.org/repos/asf/lucene/solr/trunk solr-nightly
Checked out revision 817722.
* Note that both are revision 817722
Now do builds:
cd lucene-nightly / ant / cd ..
cd solr-nightly / ant / cd ..
* OK, fine, no clover, I guess I'll live...
Now check the two Lucene core jar files:
proj $ ls -l lucene-nightly/build/lucene-core-2.9.jar
-rw-r--r-- 1 xyz staff 1104037 Sep 22 09:44
lucene-nightly/build/lucene-core-2.9.jar
proj $ ls -l solr-nightly/lib/lucene-core-2.9.0.jar
-rw-r--r-- 1 xyz staff 1104049 Sep 22 09:43
solr-nightly/lib/lucene-core-2.9.0.jar
* Notice the sizes. They're pretty close, but not identical. The jar in
lucene-nightly was created by my tools, whereas the lucene jar in Solr was
created on another machine.
To prove this use Subversion status command:
proj $ svn status lucene-nightly/build/lucene-core-2.9.jar
svn: warning: 'lucene-nightly/build/lucene-core-2.9.jar' is not a working
copy
So... not under version control, because it's built when I run ant.
proj $ svn status solr-nightly/lib/lucene-core-2.9.0.jar
(no output)
No complaints from Subversion, so we know that it IS under version control
and therefore came from the apache, and I haven't messed with it.
So I'd chalk up the difference exclusively to that then?
If folks work with both code trees a lot, maybe having a parent build file
could copy over the fresh Lucene jar over to Solr. Also curious if there's
an automated way to get this working in Eclipse.
--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
Re: Code sync between Lucene and Solr, crossing Apache project
boundaries, etc.
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, Sep 22, 2009 at 11:24 PM, Mark Bennett <mb...@ideaeng.com> wrote:
> To do any serious Solr debugging (or filter development) you also need the
> Solr source code tree. And you'd like them to be in sync, so that the
> Lucene code you see is exactly the same as what was used for the Solr
> version you're working with.
>
> I did find this link on sync'ing the two source tress, for a specific Solr
> version. But this seems a bit convoluted, is this really the right answer?
> http://happygiraffe.net/blog/2009/07/16/solrs-lucene-source/
>
> Generally:
>
> 1: Given that many threads on the Solr mailing list seem to end with "yeah,
> you could write some code to do that", it seems like most developers would
> face this problem, and yet I haven't seen it discussed much (at least not
> with the search terms I'm using, I keep finding stuff about SolrJ or just
> getting the Solr code to build)
>
> 2: Do most people pick a version of Solr, then try carefully to sync it
> with
> the exact version of Lucene code, or do they just not worry about the
> version stuff unless they get an error?
>
3: Or do most Solr/Lucene developers just live out of the Apache nightly
> snapshots for both source code trees? And if so, I have a question about
> even doing that. (see below)
>
>
Actually, I have had to debug through to Lucene sources only a couple of
times till now. I keep both Lucene trunk handy for reference. I rarely need
it if I'm writing a Solr plugin. In my IDE I setup Solr trunk with a
dependency on Lucene trunk so that I can refer to both sources by ctrl+click
on a Lucene class name. It works well because Lucene has excellent
back-compat. Before creating a patch, I run tests through ant which uses the
bundled Lucene jars.
> 4: Getting both projects to live together under Eclipse has seemed a bit
> awkward, though again I'd think it's a common task. Any good links on
> that?
>
>
I use IDEA now but I did use eclipse earlier and I had set it up the same
way. Keep both projects, from Solr remove the dependency on bundled lucene
jars, add a dependency on Lucene source for ease in cross-referencing
sources.
> 5: So do you most of you:
> a: Bother with both code trees? (Solr and Lucene)
>
Yes!
> b: Live from the command line with ant? or
>
Yes!
> c: Get them both living under Eclipse? With dependencies back and forth?
> I
> haven't found a good resource for this yet.
> d: Do you also think maven, git and clover are required?
>
Not really.
> 6: Since advanced developers using Nutch and Solr would typically want
> Lucene's source code as well, wouldn't it be good to have separate
> distribution that includes all of those? Or perhaps there's some "apache
> thing" that makes this so trivial it's not worth bothering with?
>
For most such developers I assume this would be a one-time setup which is
used for a long time. Therefore, there is very little advantage in providing
a separate distribution I guess.
> Back to question 3, just using the nightly trees for both Solr and Lucene,
> which presumably are in sync. If you don't need a specific release, this
> might be a reasonable workaround, but still a few of the details bother me.
>
> Get the code:
> svn co http://svn.apache.org/repos/asf/lucene/java/trunk lucene-nightly
> Checked out revision 817722.
> svn co http://svn.apache.org/repos/asf/lucene/solr/trunk solr-nightly
> Checked out revision 817722.
>
> * Note that both are revision 817722
>
> Now do builds:
> cd lucene-nightly / ant / cd ..
> cd solr-nightly / ant / cd ..
>
> * OK, fine, no clover, I guess I'll live...
>
> Now check the two Lucene core jar files:
> proj $ ls -l lucene-nightly/build/lucene-core-2.9.jar
> -rw-r--r-- 1 xyz staff 1104037 Sep 22 09:44
> lucene-nightly/build/lucene-core-2.9.jar
> proj $ ls -l solr-nightly/lib/lucene-core-2.9.0.jar
> -rw-r--r-- 1 xyz staff 1104049 Sep 22 09:43
> solr-nightly/lib/lucene-core-2.9.0.jar
>
> * Notice the sizes. They're pretty close, but not identical. The jar in
> lucene-nightly was created by my tools, whereas the lucene jar in Solr was
> created on another machine.
>
>
Well, trunk changes all the time. If you want exactly the same code, it is
best to checkout the Lucene revision mentioned in Solr's CHANGES.txt
--
Regards,
Shalin Shekhar Mangar.
Re: Code sync between Lucene and Solr, crossing Apache project boundaries, etc.
Posted by Grant Ingersoll <gs...@apache.org>.
On Sep 22, 2009, at 1:54 PM, Mark Bennett wrote:
> To do any serious Solr debugging (or filter development) you also
> need the
> Solr source code tree. And you'd like them to be in sync, so that the
> Lucene code you see is exactly the same as what was used for the Solr
> version you're working with.
>
> I did find this link on sync'ing the two source tress, for a
> specific Solr
> version. But this seems a bit convoluted, is this really the right
> answer?
> http://happygiraffe.net/blog/2009/07/16/solrs-lucene-source/
>
> Generally:
>
> 1: Given that many threads on the Solr mailing list seem to end with
> "yeah,
> you could write some code to do that", it seems like most developers
> would
> face this problem, and yet I haven't seen it discussed much (at
> least not
> with the search terms I'm using, I keep finding stuff about SolrJ or
> just
> getting the Solr code to build)
>
> 2: Do most people pick a version of Solr, then try carefully to sync
> it with
> the exact version of Lucene code, or do they just not worry about the
> version stuff unless they get an error?
I'm probably atypical, but I check out the exact revision number of
Lucene and associate it as source in IntelliJ. Other times, I simply
rely on the nightly Maven artifacts to get me all the correct versions
of Lucene and Solr.
>
> 3: Or do most Solr/Lucene developers just live out of the Apache
> nightly
> snapshots for both source code trees? And if so, I have a question
> about
> even doing that. (see below)
>
> 4: Getting both projects to live together under Eclipse has seemed a
> bit
> awkward, though again I'd think it's a common task. Any good links
> on that?
>
> 5: So do you most of you:
> a: Bother with both code trees? (Solr and Lucene)
Yes
> b: Live from the command line with ant? or
Yes
> c: Get them both living under Eclipse? With dependencies back and
> forth? I
> haven't found a good resource for this yet.
> d: Do you also think maven, git and clover are required?
No, they are not, but I sometimes find it easier to get up and running
in both Solr and Lucene by using the Maven artifacts.
>
> 6: Since advanced developers using Nutch and Solr would typically want
> Lucene's source code as well, wouldn't it be good to have separate
> distribution that includes all of those? Or perhaps there's some
> "apache
> thing" that makes this so trivial it's not worth bothering with?
Not sure, there hasn't been too much cross fertilization between the
projects.
>
> Back to question 3, just using the nightly trees for both Solr and
> Lucene,
> which presumably are in sync. If you don't need a specific release,
> this
> might be a reasonable workaround, but still a few of the details
> bother me.
>
> Get the code:
> svn co http://svn.apache.org/repos/asf/lucene/java/trunk lucene-
> nightly
> Checked out revision 817722.
> svn co http://svn.apache.org/repos/asf/lucene/solr/trunk solr-nightly
> Checked out revision 817722.
>
> * Note that both are revision 817722
>
I don't think this is correct. For the Lucene version, you need to
get the exact rev that Solr is expecting. This can be seen in the
Admin info page or by looking in the Manifest.MF in the Jars or in the
CHANGES.txt file.
> Now do builds:
> cd lucene-nightly / ant / cd ..
> cd solr-nightly / ant / cd ..
>
> * OK, fine, no clover, I guess I'll live...
>
> Now check the two Lucene core jar files:
> proj $ ls -l lucene-nightly/build/lucene-core-2.9.jar
> -rw-r--r-- 1 xyz staff 1104037 Sep 22 09:44
> lucene-nightly/build/lucene-core-2.9.jar
> proj $ ls -l solr-nightly/lib/lucene-core-2.9.0.jar
> -rw-r--r-- 1 xyz staff 1104049 Sep 22 09:43
> solr-nightly/lib/lucene-core-2.9.0.jar
>
> * Notice the sizes. They're pretty close, but not identical. The
> jar in
> lucene-nightly was created by my tools, whereas the lucene jar in
> Solr was
> created on another machine.
>
> To prove this use Subversion status command:
> proj $ svn status lucene-nightly/build/lucene-core-2.9.jar
> svn: warning: 'lucene-nightly/build/lucene-core-2.9.jar' is not a
> working
> copy
> So... not under version control, because it's built when I run ant.
> proj $ svn status solr-nightly/lib/lucene-core-2.9.0.jar
> (no output)
> No complaints from Subversion, so we know that it IS under version
> control
> and therefore came from the apache, and I haven't messed with it.
>
> So I'd chalk up the difference exclusively to that then?
>
> If folks work with both code trees a lot, maybe having a parent
> build file
> could copy over the fresh Lucene jar over to Solr. Also curious if
> there's
> an automated way to get this working in Eclipse.
I have a little script to copy over the correct things from the Lucene
build somewhere in JIRA, but it's not really a big deal to do it by
hand.
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search