You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by arysin <gi...@git.apache.org> on 2017/04/14 00:14:01 UTC
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
GitHub user arysin opened a pull request:
https://github.com/apache/lucene-solr/pull/187
LUCENE-7785: Move dictionary for Ukrainan analyzer to external dependency
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arysin/lucene-solr master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/lucene-solr/pull/187.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #187
----
commit f7291006b378e11cb67d9078efe52e9ace9e3f47
Author: Andriy Rysin <ar...@gmail.com>
Date: 2017-03-25T22:15:56Z
LUCENE-7785: Move dictionary for Ukrainan analyzer to external dependency
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by dweiss <gi...@git.apache.org>.
Github user dweiss commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/187#discussion_r111553388
--- Diff: lucene/analysis/morfologik/src/java/org/apache/lucene/analysis/uk/UkrainianMorfologikAnalyzer.java ---
@@ -107,11 +107,18 @@ public UkrainianMorfologikAnalyzer(CharArraySet stopwords, CharArraySet stemExcl
@Override
protected Reader initReader(String fieldName, Reader reader) {
NormalizeCharMap.Builder builder = new NormalizeCharMap.Builder();
+ // different apostrophes
builder.add("\u2019", "'");
+ builder.add("\u0218", "'");
builder.add("\u02BC", "'");
+ builder.add("`", "'");
+ builder.add("�", "'");
+ // ignored characters
builder.add("\u0301", "");
- NormalizeCharMap normMap = builder.build();
+ builder.add("\u00AD", "");
+ builder.add("\uFEFF", "");
--- End diff --
byte order mark shouldn't be replaced to nothing... if you have a byte order mark in your character input (reader) then your conversion from bytes is screwed up somewhere before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by dweiss <gi...@git.apache.org>.
Github user dweiss commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/187#discussion_r111922455
--- Diff: lucene/licenses/morfologik-ukrainian-search-NOTICE.txt ---
@@ -0,0 +1,6 @@
+morfologik-ukrainian-search is a POS tag dictionary in morfolgik format adjusted for searching.
--- End diff --
typo in morfolgik
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by sarowe <gi...@git.apache.org>.
Github user sarowe commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/187#discussion_r111573637
--- Diff: lucene/analysis/morfologik/ivy.xml ---
@@ -25,6 +25,7 @@
<dependency org="org.carrot2" name="morfologik-polish" rev="${/org.carrot2/morfologik-polish}" conf="compile"/>
<dependency org="org.carrot2" name="morfologik-fsa" rev="${/org.carrot2/morfologik-fsa}" conf="compile"/>
<dependency org="org.carrot2" name="morfologik-stemming" rev="${/org.carrot2/morfologik-stemming}" conf="compile"/>
+ <dependency org="ua.net.nlp" name="morfologik-ukrainian-search" rev="${ua.net.nlp.morfologik-ukrainian-search.version}" conf="compile"/>
--- End diff --
The `rev` attribute value has to be of the form `"${/org/name}"` - this will not pass precommit. You have to make such an entry in `ivy-versions.properties` - look at all the other examples there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by dweiss <gi...@git.apache.org>.
Github user dweiss commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/187#discussion_r111553610
--- Diff: lucene/analysis/morfologik/src/java/org/apache/lucene/analysis/uk/UkrainianMorfologikAnalyzer.java ---
@@ -145,7 +152,7 @@ protected TokenStreamComponents createComponents(String fieldName) {
private static Dictionary getDictionary() {
try {
- return Dictionary.read(UkrainianMorfologikAnalyzer.class.getResource("ukrainian.dict"));
+ return Dictionary.read(UkrainianMorfologikAnalyzer.class.getResource("/ua/net/nlp/ukrainian.dict"));
--- End diff --
This shouldn't be a class-relative read looking for an absolute resource. Rather, I think, it should be classloader-relative UkrainianMorfologikAnalyzer.class.getClassLoader().getResource("ua/net/nlp/ukrainian.dict").
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request: Lucene solr 5 2
Posted by aniketkhare <gi...@git.apache.org>.
Github user aniketkhare closed the pull request at:
https://github.com/apache/lucene-solr/pull/187
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by sarowe <gi...@git.apache.org>.
Github user sarowe commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/187#discussion_r111574446
--- Diff: lucene/ivy-versions.properties ---
@@ -203,6 +203,8 @@ org.carrot2.morfologik.version = 2.1.1
/org.carrot2/morfologik-polish = ${org.carrot2.morfologik.version}
/org.carrot2/morfologik-stemming = ${org.carrot2.morfologik.version}
+ua.net.nlp.morfologik-ukrainian-search.version = 3.7.4
--- End diff --
This is fine, but you're missing the `/org/name=version` thing that's required to be used in `ivy.xml` files.
If you look at the rest of this file, you can see that some `/org/name` keys share the same version, and so a variable is introduced to simplify maintenance. In your case, when there is only one `/org/name` key using this version, you should IMO eliminate the `ua.net.nlp.morfologik-ukrainian-search.version` variable entirely and add `/ua.net.nlp/morfologik-ukrainian-search = 3.7.4`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr issue #187: LUCENE-7785: Move dictionary for Ukrainan analyzer t...
Posted by arysin <gi...@git.apache.org>.
Github user arysin commented on the issue:
https://github.com/apache/lucene-solr/pull/187
It would be also nice to merge this into 6x branch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr issue #187: LUCENE-7785: Move dictionary for Ukrainan analyzer t...
Posted by arysin <gi...@git.apache.org>.
Github user arysin commented on the issue:
https://github.com/apache/lucene-solr/pull/187
Thanks for the prompt feedback guys, I think I've addressed all the issues with the latest commits but please let me know if something is still wrong.
Once we're good with master branch I'll prepare the 6x changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by arysin <gi...@git.apache.org>.
Github user arysin commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/187#discussion_r111596732
--- Diff: lucene/analysis/morfologik/src/java/org/apache/lucene/analysis/uk/UkrainianMorfologikAnalyzer.java ---
@@ -145,7 +152,7 @@ protected TokenStreamComponents createComponents(String fieldName) {
private static Dictionary getDictionary() {
try {
- return Dictionary.read(UkrainianMorfologikAnalyzer.class.getResource("ukrainian.dict"));
+ return Dictionary.read(UkrainianMorfologikAnalyzer.class.getResource("/ua/net/nlp/ukrainian.dict"));
--- End diff --
I am a bit confused on why for class loader getResource() we need relative path (surprisingly the full one starting with "/" does not work for me), but otherwise it makes sense so I'll push the fix shortly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by arysin <gi...@git.apache.org>.
Github user arysin commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/187#discussion_r111595706
--- Diff: lucene/analysis/morfologik/src/java/org/apache/lucene/analysis/uk/UkrainianMorfologikAnalyzer.java ---
@@ -107,11 +107,18 @@ public UkrainianMorfologikAnalyzer(CharArraySet stopwords, CharArraySet stemExcl
@Override
protected Reader initReader(String fieldName, Reader reader) {
NormalizeCharMap.Builder builder = new NormalizeCharMap.Builder();
+ // different apostrophes
builder.add("\u2019", "'");
+ builder.add("\u0218", "'");
builder.add("\u02BC", "'");
+ builder.add("`", "'");
+ builder.add("�", "'");
+ // ignored characters
builder.add("\u0301", "");
- NormalizeCharMap normMap = builder.build();
+ builder.add("\u00AD", "");
+ builder.add("\uFEFF", "");
--- End diff --
That was from the note [Wikimedia guys suggested](https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Ukrainian_Morfologik_Analysis#Recommendations_.26_Plan), but agree it does not make sense here, I'll remove it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by dweiss <gi...@git.apache.org>.
Github user dweiss commented on a diff in the pull request:
https://github.com/apache/lucene-solr/pull/187#discussion_r111922745
--- Diff: lucene/licenses/morfologik-ukrainian-search-NOTICE.txt ---
@@ -0,0 +1,6 @@
+morfologik-ukrainian-search is a POS tag dictionary in morfolgik format adjusted for searching.
+It's part of dict_uk project (https://github.com/brown-uk/dict_uk)
+
+Note: to better fit into full-text search model this dictionary has all word forms in lower case but keeps lemmas for proper nouns in upper case.
+
+Licensed under GPL/LGPL, CC BY-NC-SA 4.0, and Apache License 2.0.
--- End diff --
Hmm; not a lawyer, but this should better state "Licensed under Apache License 2.0"... Those multiple licenses don't agree with each other -- at least it should state *or*, not *and*.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[GitHub] lucene-solr pull request #187: LUCENE-7785: Move dictionary for Ukrainan ana...
Posted by arysin <gi...@git.apache.org>.
Github user arysin closed the pull request at:
https://github.com/apache/lucene-solr/pull/187
---
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org