You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Daniel Naber <da...@t-online.de> on 2004/08/14 19:34:09 UTC

moving the analyzers into sandbox

Hi,

any objections against moving the German and Russian analyzers into the 
sandbox? If not, I'd like to do that, but I'm not sure if we already 
agreed on doing so. The current situation with analyzers both in lucene's 
core and in the sandbox doesn't seem to make sense.

I suggest that we provide an lucene-analyzers.tar.gz with the upcoming 
releases so that people don't have to check out the sandbox from CVS to 
make use of the analyzers.

Regards
 Daniel

-- 
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 15, 2004, at 8:00 AM, Daniel Naber wrote:
> On Saturday 14 August 2004 22:44, Erik Hatcher wrote:
>
>> +1 on moving those analyzers out.
>
> Technically this means to remove the files from lucene's core and 
> re-add
> them in sandbox, doesn't it? Or is there some special case for moving
> (e.g. contact cvs admin so he can move files on the server)?

Right, we just add them to the sandbox and remove them from the core.  
CVS doesn't support "move" operations.  We could, though, (and this is 
a separate discussion altogether) move to Subversion.  Several Apache 
projects already have and many more are going that direction.

> I agree. Some more ideas how we can improve the release process:
>
> -make sure the mirrors get the new version (don't know what's required 
> for
> that)

We don't even  have 1.4.1 mirrored yet.  We need to be sure to make 
that happen soon though.  I've done it for the past couple of releases. 
  It's not that big of a deal, but does involve several manual steps 
including signing each of the files locally and uploading them into the 
correct spot at www.apache.org.  There are instructions at the old 
nagoya wiki somewhere and I have them archived in e-mail.

> I could create a small document that lists the things to do for each
> release. If you think that such a document is useful, should I put it 
> into
> cvs or in the wiki?

I think the wiki is the best place.  Most of the instructions are 
actually out there, but they are not Lucene specific:

	http://nagoya.apache.org/wiki/apachewiki.cgi?ReleaseManager

Having the steps on the Lucene wiki would be nice though.

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Daniel Naber <da...@t-online.de>.
On Saturday 14 August 2004 22:44, Erik Hatcher wrote:

> +1 on moving those analyzers out.

Technically this means to remove the files from lucene's core and re-add 
them in sandbox, doesn't it? Or is there some special case for moving 
(e.g. contact cvs admin so he can move files on the server)?

> > I suggest that we provide an lucene-analyzers.tar.gz with the upcoming
> > releases so that people don't have to check out the sandbox from CVS
> > to make use of the analyzers.
>
> We should go even further and release all sandbox components along with
> each release - this would give more exposure to the highlighter,
> snowball analyzer, and the others.

I agree. Some more ideas how we can improve the release process:

-Make sure the new version gets properly announced:
	-on annoucements@jakarta.apache.org
	-on freshmeat.net and maybe other services
-Make sure Bugzilla gets the new version number so people can select it 
when they submit a bug report (1.4.1 is still missing)
-make sure the mirrors get the new version (don't know what's required for 
that)

I could create a small document that lists the things to do for each 
release. If you think that such a document is useful, should I put it into 
cvs or in the wiki?

Regards
 Daniel

-- 
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 16, 2004, at 5:15 PM, Daniel Naber wrote:
> On Sunday 15 August 2004 13:42, Otis Gospodnetic wrote:
>
>> +1 for releasing at least some Sandbox components.  Analyzers, 
>> Snowball
>> Analyzers and Highligher at least.
>>
>> Is this something you can/want to do, Daniel?
>
> The analyzers are now in the sandbox

Nice job moving things around.  Be sure to update the CHANGES.txt file 
with this info, especially with the info on changing the word list 
loader package.  That change may break some folks that are using that 
class externally (although I'm personally fine with breaking BWC when 
it is for the betterment of the codebase).

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Doug Cutting <cu...@apache.org>.
Erik Hatcher wrote:
> On Aug 16, 2004, at 5:15 PM, Daniel Naber wrote:
> 
>> On Sunday 15 August 2004 13:42, Otis Gospodnetic wrote:
>>
>>> +1 for releasing at least some Sandbox components.  Analyzers, Snowball
>>> Analyzers and Highligher at least.
>>>
>>> Is this something you can/want to do, Daniel?
>>
>> The analyzers are now in the sandbox, I'll leave it to our ant expert to
>> modify/create the appropriate build.xml for the other components...
> 
> What is desired that currently isn't available with the top-level 
> contributions directory build.xml?

+1 for releasing the sandbox components with the core.

But how should we do this?  Should we bundle everything into one big 
lucene-X.X.tar.gz file?  Or should we have two files, 
lucene-core-X.X.tar.gz and lucene-contrib-X.X.tar.gz?  Or should we keep 
the contributions each packaged separately, e.g. lucene-core-X.X.tar.gz, 
lucene-snowball-X.X.tar.gz, etc.  Or should we put all of the 
contributions in a subdirectory, e.g., lucene-X.X.tar.gz, 
contrib/lucene-snowball-X.X.tar.gz, etc.  I prefer one of the latter, 
but not strongly.

Whatever the case, I think we should have a single command line that 
builds both the core and all of the sandbox contributions, runs unit 
tests on everything, and places them all in single distribution 
directory.  If I'm not mistaken, I think that's the bit of ant wizardry 
that Daniel was referring to.

In the end, I think the distinction between core and sandbox should be 
less one of quality and more one of packaging.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 17, 2004, at 10:11 AM, Daniel Naber wrote:
> On Tuesday 17 August 2004 14:45, Erik Hatcher wrote:
>
>> src/java/org/apache/lucene/analysis/fr/FrenchStemmer.java:571:  
>> duplicate case label
>>      [javac]                     case '?':
>>      [javac]                         ^
>
> "ant compile" works for me, probably because my system defaults to  
> utf-8.
> But when I explicitly open the FrenchStemmer.java as utf-8, the special
> characters are okay, so it's in utf-8, just like GermanStemmer.java.  
> So it
> should compile when the encoding is set to utf-8.

Running "ant compile -verbose", I get this:


    [javac]  
'/Users/erik/dev/jakarta/jakarta-lucene-sandbox/contributions/ 
analyzers/src/java'
     [javac] '-encoding'
     [javac] 'utf-8'
     [javac] '-g'
     [javac]

...

     [javac]  
/Users/erik/dev/jakarta/jakarta-lucene-sandbox/contributions/analyzers/ 
src/java/org/apache/lucene/analysis/fr/FrenchStemmer.java:571:  
duplicate case label
     [javac]                     case '?':
     [javac]                         ^


And same type of errors with the DutchStemmer.  I'm doing this on Mac  
OS X.

I've done a fresh checkout of all "fr" files to no avail.

If someone can help me past compile errors, I'll gladly work on the  
sandbox packaging :)

	Erik

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Daniel Naber <da...@t-online.de>.
On Tuesday 17 August 2004 14:45, Erik Hatcher wrote:

> src/java/org/apache/lucene/analysis/fr/FrenchStemmer.java:571:  
> duplicate case label
>      [javac]                     case '?':
>      [javac]                         ^

"ant compile" works for me, probably because my system defaults to utf-8. 
But when I explicitly open the FrenchStemmer.java as utf-8, the special 
characters are okay, so it's in utf-8, just like GermanStemmer.java. So it 
should compile when the encoding is set to utf-8.

Regards
 Daniel

-- 
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 16, 2004, at 5:32 PM, Daniel Naber wrote:
> However, "ant package" currently
> doesn't work in contributions/analyzers (first it cannot find an
> overview.html and if you add that it complains that
> contributions/analyzers/docs cannot be found).

I'm getting more serious errors with the analyzers project.  Here is  
what I see:

compile:
     [javac] Compiling 26 source files to  
/Users/erik/dev/jakarta/jakarta-lucene-sandbox/contributions/analyzers/ 
build/classes
     [javac]  
/Users/erik/dev/jakarta/jakarta-lucene-sandbox/contributions/analyzers/ 
src/java/org/apache/lucene/analysis/fr/FrenchStemmer.java:571:  
duplicate case label
     [javac]                     case '?':
     [javac]                         ^


I've modified ../common.xml to have a build.encoding="UTF-8" and  
modified the <javac>'s in there to use it (all locally) which got me  
past the compilation issues of the GermanStemmer.  I'm guessing the  
FrenchStemmer is in a different encoding.

What needs to be done to alleviate this?

Thanks,
	Erik

>
> Regards
>  Daniel
>
> -- 
> http://www.danielnaber.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Daniel Naber <da...@t-online.de>.
On Monday 16 August 2004 23:16, Erik Hatcher wrote:

> What is desired that currently isn't available with the top-level
> contributions directory build.xml?
>
> The build packages each project into a .tar.gz including javadocs, and
> runs unit tests too.

I see. Then probably nothing is missing. However, "ant package" currently 
doesn't work in contributions/analyzers (first it cannot find an 
overview.html and if you add that it complains that 
contributions/analyzers/docs cannot be found).

Regards
 Daniel

-- 
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 16, 2004, at 5:15 PM, Daniel Naber wrote:
> On Sunday 15 August 2004 13:42, Otis Gospodnetic wrote:
>
>> +1 for releasing at least some Sandbox components.  Analyzers, 
>> Snowball
>> Analyzers and Highligher at least.
>>
>> Is this something you can/want to do, Daniel?
>
> The analyzers are now in the sandbox, I'll leave it to our ant expert 
> to
> modify/create the appropriate build.xml for the other components...

What is desired that currently isn't available with the top-level 
contributions directory build.xml?

The build packages each project into a .tar.gz including javadocs, and 
runs unit tests too.  I'd be happy to tinker with it if there is 
something missing.

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Daniel Naber <da...@t-online.de>.
On Sunday 15 August 2004 13:42, Otis Gospodnetic wrote:

> +1 for releasing at least some Sandbox components.  Analyzers, Snowball
> Analyzers and Highligher at least.
>
> Is this something you can/want to do, Daniel?

The analyzers are now in the sandbox, I'll leave it to our ant expert to 
modify/create the appropriate build.xml for the other components...

Regards
 Daniel

-- 
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Otis Gospodnetic <ot...@yahoo.com>.
+1 for moving analyzers to the sandbox.  We've talked about this
before, and I believe that is what we concluded with.

+1 for releasing at least some Sandbox components.  Analyzers, Snowball
Analyzers and Highligher at least.

Is this something you can/want to do, Daniel?

Otis


--- Erik Hatcher <er...@ehatchersolutions.com> wrote:

> 
> On Aug 14, 2004, at 1:34 PM, Daniel Naber wrote:
> > any objections against moving the German and Russian analyzers into
> the
> > sandbox? If not, I'd like to do that, but I'm not sure if we
> already
> > agreed on doing so. The current situation with analyzers both in 
> > lucene's
> > core and in the sandbox doesn't seem to make sense.
> 
> +1 on moving those analyzers out.
> 
> > I suggest that we provide an lucene-analyzers.tar.gz with the
> upcoming
> > releases so that people don't have to check out the sandbox from
> CVS to
> > make use of the analyzers.
> 
> We should go even further and release all sandbox components along
> with 
> each release - this would give more exposure to the highlighter, 
> snowball analyzer, and the others.  Maybe we shouldn't package 
> everything that way, as some things might tend to cause more support 
> issues, such as WordNet which requires a few additional steps to make
> 
> use of.  Thoughts?  At the very least though, analyzers, highlighter,
> 
> and snowball make great sense to package and version with Lucene core
> 
> releases, I think.
> 
> 	Erik
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: moving the analyzers into sandbox

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Aug 14, 2004, at 1:34 PM, Daniel Naber wrote:
> any objections against moving the German and Russian analyzers into the
> sandbox? If not, I'd like to do that, but I'm not sure if we already
> agreed on doing so. The current situation with analyzers both in 
> lucene's
> core and in the sandbox doesn't seem to make sense.

+1 on moving those analyzers out.

> I suggest that we provide an lucene-analyzers.tar.gz with the upcoming
> releases so that people don't have to check out the sandbox from CVS to
> make use of the analyzers.

We should go even further and release all sandbox components along with 
each release - this would give more exposure to the highlighter, 
snowball analyzer, and the others.  Maybe we shouldn't package 
everything that way, as some things might tend to cause more support 
issues, such as WordNet which requires a few additional steps to make 
use of.  Thoughts?  At the very least though, analyzers, highlighter, 
and snowball make great sense to package and version with Lucene core 
releases, I think.

	Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org