You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jim Regan (Created) (JIRA)" <ji...@apache.org> on 2012/03/19 17:12:39 UTC
[jira] [Created] (LUCENE-3883) Analysis for Irish
Analysis for Irish
------------------
Key: LUCENE-3883
URL: https://issues.apache.org/jira/browse/LUCENE-3883
Project: Lucene - Java
Issue Type: New Feature
Components: modules/analysis
Reporter: Jim Regan
Priority: Trivial
Adds analysis for Irish.
The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3883:
--------------------------------
Attachment: LUCENE-3883.patch
updated patch, with a simple solution to the hyphen-phrasequery-problem:
I added a special stopset just for these:
{code}
/**
* When StandardTokenizer splits t‑athair into {t, athair}, we don't
* want to cause a position increment, otherwise there will be problems
* with phrase queries versus tAthair (which would not have a gap).
*/
private static final CharArraySet HYPHENATIONS = CharArraySet.unmodifiableSet(
new CharArraySet(Version.LUCENE_CURRENT,
Arrays.asList(
"h", "n", "t"
), true));
{code}
This is used with enablePositionIncrements=false to ensure no gap is added... I also added a simple test for this.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232721#comment-13232721 ]
Jim Regan commented on LUCENE-3883:
-----------------------------------
Yeah, it's quite an odd thing (Scots Gaelic has a similar phenomenon, but they consistently keep the hyphen), but it does help with the stemmer in those cases to know that the t or n at the start of the word is due only to mutation.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233470#comment-13233470 ]
Robert Muir commented on LUCENE-3883:
-------------------------------------
Thanks Jim. Personally I think this patch is ready to be committed.
I'm just going to wait a bit in case you get any feedback from Martin or other snowball developers,
but I won't wait too long :)
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271783#comment-13271783 ]
Jim Regan commented on LUCENE-3883:
-----------------------------------
Just to follow up, the Irish stemmer is now available from the Snowball site: http://snowball.tartarus.org/otherapps/oregan/intro.html
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "David Smiley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233621#comment-13233621 ]
David Smiley commented on LUCENE-3883:
--------------------------------------
How ironic this issue is created nearly on St. Patrick's Day.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233593#comment-13233593 ]
Jim Regan commented on LUCENE-3883:
-----------------------------------
Great :)
Regarding the initial 'h', I asked Kevin Scannell (among other feathers in his cap, he created the dictionary used in GaelSpell, and ran an Irish-language search engine), who said:
"I looked carefully at how often initial h is a prefix vs not a while ago. I can send you those data - non-prefixes might be more common than you'd think in running text bc of proper names, English mixed in, etc. So upshot is it's a bad idea to strip all initial h's with no hyphen following.
As far as h- (with hyphen) goes, it's non-standard but common enough that I'd leave it in the stemmer. Not like there would be false positives in that case if the hyphen is there.'
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271831#comment-13271831 ]
Jim Regan commented on LUCENE-3883:
-----------------------------------
I wouldn't recommend the aggressive mode, and I regret that I left it uncommented. If you really think an alternative would be welcome, it would be quite easy to get the best of both (in fact, I spent roughly half the time on that trying to beat Snowball into overstemming to match the original).
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237599#comment-13237599 ]
Jim Regan commented on LUCENE-3883:
-----------------------------------
Yay! Thanks for all your help!
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271792#comment-13271792 ]
Robert Muir commented on LUCENE-3883:
-------------------------------------
Thanks Jim! I already removed our local copy of the irish.sbl as its now available on
the snowball site.
I have to investigate the Czech implementation, I think we should make it available
as well, since it also supports stemming of derivational endings: Dawid opened
LUCENE-4042 for that.
Thanks for contributing these to snowball.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233169#comment-13233169 ]
Robert Muir commented on LUCENE-3883:
-------------------------------------
Hmm another downside of #1 is that with a simple stopfilter approach, position increments won't line up
if we have a phrase query of "n-athair" with indexed nAthair.
So I start to lean towards #2 since it would be a better solution... but I'm going to think about it
and see if I come up with any other ideas.
Separately, what about h- when succeeded by a vowel? Is there actually usually a hyphen here?
(Wikipedia says no, playing around with GaelSpell seems to agree, but I don't know anything about this language!)
Would this case be too aggressive to handle?
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Regan updated LUCENE-3883:
------------------------------
Attachment: LUCENE-3883.patch
New version of patch, also checking that chLen (array length) > 1
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233632#comment-13233632 ]
Jim Regan commented on LUCENE-3883:
-----------------------------------
It was on my mind, a little :) I made the stemmer on the 15th, on the 17th I made ICU transliteration rules for Irish->IPA, but that's not quite relevant here.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233465#comment-13233465 ]
Jim Regan commented on LUCENE-3883:
-----------------------------------
Wow! Thanks Robert!
There isn't usually a hyphen with 'h' before a vowel, but I've started to see it recently -- there are no native Irish words beginning with 'h', so it used to be relatively unambiguous that a 'h' was a mutation, but with an increase of scientific literature in Irish, there are more Greek and Latin loan words being added which do begin with 'h', so it's no longer clear.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233159#comment-13233159 ]
Robert Muir commented on LUCENE-3883:
-------------------------------------
Thanks for updating the patch Jim!
one concern doing some very very rudimentary testing:
we have special lowercasing for situations like nAthair -> n-athair,
which the snowball rules then strip:
{noformat}
define initial_morph as (
[substring] among (
'h-' 'n-' 't-' //nAthair -> n-athair, but alone are problematic
(delete)
{noformat}
The problem is if the input initially comes as n-athair, Unicode break rules
will split this up on the hyphen into two tokens {n, athair}. You can visualize this at http://unicode.org/cldr/utility/breaks.jsp
This means we can add many spurious 'n' tokens in the index...
So we have two potential solutions to this:
# we can simply add 'n', 'h', 't', etc to the stopwords list. This is the simplest solution. Would this be too aggressive?
# we can add a CharFilter for IrishAnalyzer to prevent this splitting from happening. This is more complex.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232717#comment-13232717 ]
Robert Muir commented on LUCENE-3883:
-------------------------------------
By the way I created LUCENE-3884 to move the ElisionFilter out of the french package
into a more general .util package. That doesnt need to hold up this issue: it just
reminded me we should move it because its not really french-specific.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3883:
--------------------------------
Attachment: LUCENE-3883.patch
Same patch but with the solr pieces too (factory/test for the lowercasefilter, text_ga fieldtype, resources synced, etc).
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233183#comment-13233183 ]
Robert Muir commented on LUCENE-3883:
-------------------------------------
To make matters worse: this exact example of splitting on hyphen for this Irish case is
actually mentioned on http://en.wikipedia.org/wiki/Hyphen#In_computing
>From there it seems like the right thing to do is heuristically convert to
U+2011 (non-breaking hyphen) but this only affects Unicode line-break rules,
not word break rules :(
So it seems like the least hackish workaround would be for a charfilter to
convert n-athair -> nAthair (to prevent the tokenizer from splitting it up),
since the IrishLowerCaseFilter will convert it back and stem it anyway.
I'll see if i can hack something up.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-3883.
---------------------------------
Resolution: Fixed
Fix Version/s: 4.0
3.6
Thank you very much Jim! I just committed this.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Regan updated LUCENE-3883:
------------------------------
Attachment: irish.sbl
Irish snowball script
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reassigned LUCENE-3883:
-----------------------------------
Assignee: Robert Muir
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Regan updated LUCENE-3883:
------------------------------
Attachment: (was: LUCENE-3883.patch)
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233597#comment-13233597 ]
Robert Muir commented on LUCENE-3883:
-------------------------------------
This makes sense to me, I agree with the conservative approach here!
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Assignee: Robert Muir
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232712#comment-13232712 ]
Uwe Schindler commented on LUCENE-3883:
---------------------------------------
Hi,
very funny lowercase filter! :-) One thing: It does not actually ArrayIndexOutOfBoundsEx in the filter because of the way how CharTermAttributeImpl is implemented internally, but theoretically there is a length check missing. The nUpper/tUpper stuff can get out of bounds if the length of term in 0 or 1 (which are valid length). But thats only a minor complaint about the code. Otherwise looks great. Just appearing from no irish support at all! really needed! :-)
Uwe
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232706#comment-13232706 ]
Robert Muir commented on LUCENE-3883:
-------------------------------------
Thanks Jim! This looks really nicely done...
Out of curiousity could you share your snowball rules (the .sbl) with us?
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Regan updated LUCENE-3883:
------------------------------
Attachment: (was: LUCENE-3883.patch)
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Regan updated LUCENE-3883:
------------------------------
Attachment: LUCENE-3883.patch
Patch, redone from top level of svn.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Regan updated LUCENE-3883:
------------------------------
Attachment: LUCENE-3883.patch
Patch adding Irish analysis
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3883) Analysis for Irish
Posted by "Jim Regan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232732#comment-13232732 ]
Jim Regan commented on LUCENE-3883:
-----------------------------------
I'm not sure if I actually needed to use the ElisionFilter, because the stemmer handles those - because of the initial mutation in Irish, trimming the start of the word is more important than trimming the end. I was copying the Catalan analyser, and using ElisionFilter seemed like The Thing To Do.
> Analysis for Irish
> ------------------
>
> Key: LUCENE-3883
> URL: https://issues.apache.org/jira/browse/LUCENE-3883
> Project: Lucene - Java
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Jim Regan
> Priority: Trivial
> Labels: analysis, newbie
> Attachments: LUCENE-3883.patch, irish.sbl
>
>
> Adds analysis for Irish.
> The stemmer is generated from a snowball stemmer. I've sent it to Martin Porter, who says it will be added during the week.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org