You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/02/27 17:59:03 UTC
[GitHub] [lucene-solr] danmuzi opened a new pull request #1296: LUCENE-9253:
Support custom dictionaries in KoreanTokenizer
danmuzi opened a new pull request #1296: LUCENE-9253: Support custom dictionaries in KoreanTokenizer
URL: https://github.com/apache/lucene-solr/pull/1296
KoreanTokenizer does not support custom dictionaries(system, unknown) now, even though Nori provides DictionaryBuilder that creates custom dictionary.
In the current state, it is very difficult for Nori users to use a custom dictionary.
Therefore, we need to open a new constructor that uses it.
JIRA : https://issues.apache.org/jira/browse/LUCENE-9253
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] danmuzi commented on a change in pull request #1296:
LUCENE-9253: Support custom dictionaries in KoreanTokenizer
Posted by GitBox <gi...@apache.org>.
danmuzi commented on a change in pull request #1296: LUCENE-9253: Support custom dictionaries in KoreanTokenizer
URL: https://github.com/apache/lucene-solr/pull/1296#discussion_r386122724
##########
File path: lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/BinaryDictionary.java
##########
@@ -150,7 +150,18 @@ protected final InputStream getResource(String suffix) throws IOException {
throw new IllegalStateException("unknown resource scheme " + resourceScheme);
}
}
-
+
+ public static InputStream getResource(ResourceScheme scheme, String path) throws IOException {
Review comment:
It's the same as Kuromoji's pattern.
But there is no relation with Kuromoji.
It's already written as an enum in this class.
https://github.com/apache/lucene-solr/blob/d5e44e95175dbf027915e162925057bbcc14200b/lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/BinaryDictionary.java#L45-L47
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] msokolov commented on a change in pull request #1296:
LUCENE-9253: Support custom dictionaries in KoreanTokenizer
Posted by GitBox <gi...@apache.org>.
msokolov commented on a change in pull request #1296: LUCENE-9253: Support custom dictionaries in KoreanTokenizer
URL: https://github.com/apache/lucene-solr/pull/1296#discussion_r386108024
##########
File path: lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/dict/BinaryDictionary.java
##########
@@ -150,7 +150,18 @@ protected final InputStream getResource(String suffix) throws IOException {
throw new IllegalStateException("unknown resource scheme " + resourceScheme);
}
}
-
+
+ public static InputStream getResource(ResourceScheme scheme, String path) throws IOException {
Review comment:
so .. this basically follows the pattern from JapaneseTokenizer, I think. .. but somehow I don't see where we defined ResourceScheme? We're not referencing the one in kuromoji, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] danmuzi commented on issue #1296: LUCENE-9253:
Support custom dictionaries in KoreanTokenizer
Posted by GitBox <gi...@apache.org>.
danmuzi commented on issue #1296: LUCENE-9253: Support custom dictionaries in KoreanTokenizer
URL: https://github.com/apache/lucene-solr/pull/1296#issuecomment-593510982
@jimczi
I'll push it to master and 8.x branch because the branch 8.5 cut date is almost up.
Please let me know if there is a problem :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] danmuzi merged pull request #1296: LUCENE-9253:
Support custom dictionaries in KoreanTokenizer
Posted by GitBox <gi...@apache.org>.
danmuzi merged pull request #1296: LUCENE-9253: Support custom dictionaries in KoreanTokenizer
URL: https://github.com/apache/lucene-solr/pull/1296
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] danmuzi commented on issue #1296: LUCENE-9253:
Support custom dictionaries in KoreanTokenizer
Posted by GitBox <gi...@apache.org>.
danmuzi commented on issue #1296: LUCENE-9253: Support custom dictionaries in KoreanTokenizer
URL: https://github.com/apache/lucene-solr/pull/1296#issuecomment-593120577
Thanks for your review! @msokolov
I changed some upper cases in Javadoc.
And I added a change log for this patch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] danmuzi commented on a change in pull request #1296:
LUCENE-9253: Support custom dictionaries in KoreanTokenizer
Posted by GitBox <gi...@apache.org>.
danmuzi commented on a change in pull request #1296: LUCENE-9253: Support custom dictionaries in KoreanTokenizer
URL: https://github.com/apache/lucene-solr/pull/1296#discussion_r386122678
##########
File path: lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/KoreanTokenizer.java
##########
@@ -185,16 +185,43 @@ public KoreanTokenizer(AttributeFactory factory, UserDictionary userDictionary,
* @param discardPunctuation true if punctuation tokens should be dropped from the output.
*/
public KoreanTokenizer(AttributeFactory factory, UserDictionary userDictionary, DecompoundMode mode, boolean outputUnknownUnigrams, boolean discardPunctuation) {
+ this(factory,
+ TokenInfoDictionary.getInstance(),
+ UnknownDictionary.getInstance(),
+ ConnectionCosts.getInstance(),
+ userDictionary, mode, outputUnknownUnigrams, discardPunctuation);
+ }
+
+ /**
+ * <p>Create a new KoreanTokenizer supplying a custom system dictionary and unknown dictionary.
+ * This constructor provides an entry point for users that want to construct custom language models
+ * that can be used as input to {@link org.apache.lucene.analysis.ko.util.DictionaryBuilder}.</p>
+ *
+ * @param factory the AttributeFactory to use
+ * @param systemDictionary a custom known token dictionary
+ * @param unkDictionary a custom unknown token dictionary
+ * @param connectionCosts custom token transition costs
+ * @param userDictionary Optional: if non-null, user dictionary.
+ * @param mode Decompound mode.
+ * @param outputUnknownUnigrams If true outputs unigrams for unknown words.
Review comment:
Oh, I did that because it was capitalized before.
I'll change the other constructors as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] msokolov commented on a change in pull request #1296:
LUCENE-9253: Support custom dictionaries in KoreanTokenizer
Posted by GitBox <gi...@apache.org>.
msokolov commented on a change in pull request #1296: LUCENE-9253: Support custom dictionaries in KoreanTokenizer
URL: https://github.com/apache/lucene-solr/pull/1296#discussion_r386107769
##########
File path: lucene/analysis/nori/src/java/org/apache/lucene/analysis/ko/KoreanTokenizer.java
##########
@@ -185,16 +185,43 @@ public KoreanTokenizer(AttributeFactory factory, UserDictionary userDictionary,
* @param discardPunctuation true if punctuation tokens should be dropped from the output.
*/
public KoreanTokenizer(AttributeFactory factory, UserDictionary userDictionary, DecompoundMode mode, boolean outputUnknownUnigrams, boolean discardPunctuation) {
+ this(factory,
+ TokenInfoDictionary.getInstance(),
+ UnknownDictionary.getInstance(),
+ ConnectionCosts.getInstance(),
+ userDictionary, mode, outputUnknownUnigrams, discardPunctuation);
+ }
+
+ /**
+ * <p>Create a new KoreanTokenizer supplying a custom system dictionary and unknown dictionary.
+ * This constructor provides an entry point for users that want to construct custom language models
+ * that can be used as input to {@link org.apache.lucene.analysis.ko.util.DictionaryBuilder}.</p>
+ *
+ * @param factory the AttributeFactory to use
+ * @param systemDictionary a custom known token dictionary
+ * @param unkDictionary a custom unknown token dictionary
+ * @param connectionCosts custom token transition costs
+ * @param userDictionary Optional: if non-null, user dictionary.
+ * @param mode Decompound mode.
+ * @param outputUnknownUnigrams If true outputs unigrams for unknown words.
Review comment:
Don't capitalize "If"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org