You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opennlp.apache.org by GitBox <gi...@apache.org> on 2022/03/25 16:31:44 UTC

[GitHub] [opennlp] jzonthemtn opened a new pull request #383: OPENNLP-1318: Add automatic model downloading.

jzonthemtn opened a new pull request #383:
URL: https://github.com/apache/opennlp/pull/383


   Thank you for contributing to Apache OpenNLP.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [X] Is there a JIRA ticket associated with this PR? Is it referenced 
        in the commit message?
   
   - [X] Does your PR title start with OPENNLP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
   
   - [X] Has your PR been rebased against the latest commit within the target branch (typically master)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [X] Have you ensured that the full suite of tests is executed via mvn clean install at the root opennlp folder?
   - [X] Have you written or updated unit tests to verify your changes?
   - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file in opennlp folder?
   - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found in opennlp folder?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [opennlp] jzonthemtn commented on a change in pull request #383: OPENNLP-1318: Add automatic model downloading.

Posted by GitBox <gi...@apache.org>.
jzonthemtn commented on a change in pull request #383:
URL: https://github.com/apache/opennlp/pull/383#discussion_r535575813



##########
File path: opennlp-tools/src/main/java/opennlp/tools/util/DownloadUtil.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+
+import opennlp.tools.chunker.ChunkerModel;
+import opennlp.tools.namefind.TokenNameFinderModel;
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+import opennlp.tools.util.model.BaseModel;
+
+public class DownloadUtil {

Review comment:
       I will add that. I'll update the user's guide, too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [opennlp] jzonthemtn merged pull request #383: OPENNLP-1318: Add automatic model downloading.

Posted by GitBox <gi...@apache.org>.
jzonthemtn merged pull request #383:
URL: https://github.com/apache/opennlp/pull/383


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [opennlp] jzonthemtn commented on pull request #383: OPENNLP-1318: Add automatic model downloading.

Posted by GitBox <gi...@apache.org>.
jzonthemtn commented on pull request #383:
URL: https://github.com/apache/opennlp/pull/383#issuecomment-738857071


   Closing until we get new models trained.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [opennlp] kinow commented on a change in pull request #383: OPENNLP-1318: Add automatic model downloading.

Posted by GitBox <gi...@apache.org>.
kinow commented on a change in pull request #383:
URL: https://github.com/apache/opennlp/pull/383#discussion_r535591013



##########
File path: opennlp-tools/src/test/java/opennlp/tools/namefind/NameFinderMETest.java
##########
@@ -109,6 +110,21 @@ public void testNameFinder() throws Exception {
     Assert.assertEquals(new Span(4, 6, DEFAULT), names[1]);
   }
 
+  @Test
+  public void testNameFinderWithDownloadedModel() throws Exception {
+
+    String input = "Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .";
+    String[] sentence = input.split(" ");
+
+    TokenNameFinder nameFinder = new NameFinderME("en", DownloadUtil.EntityType.PERSON);

Review comment:
       Good idea, or group them so that if one starts failing we can disable temporarily until whatever issue is fixed and they can be re-enabled.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [opennlp] jzonthemtn commented on a change in pull request #383: OPENNLP-1318: Add automatic model downloading.

Posted by GitBox <gi...@apache.org>.
jzonthemtn commented on a change in pull request #383:
URL: https://github.com/apache/opennlp/pull/383#discussion_r535576224



##########
File path: opennlp-tools/src/test/java/opennlp/tools/namefind/NameFinderMETest.java
##########
@@ -109,6 +110,21 @@ public void testNameFinder() throws Exception {
     Assert.assertEquals(new Span(4, 6, DEFAULT), names[1]);
   }
 
+  @Test
+  public void testNameFinderWithDownloadedModel() throws Exception {
+
+    String input = "Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .";
+    String[] sentence = input.split(" ");
+
+    TokenNameFinder nameFinder = new NameFinderME("en", DownloadUtil.EntityType.PERSON);

Review comment:
       That's a good point. I'll see if I can make those integration tests that aren't run with every build.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [opennlp] kinow commented on a change in pull request #383: OPENNLP-1318: Add automatic model downloading.

Posted by GitBox <gi...@apache.org>.
kinow commented on a change in pull request #383:
URL: https://github.com/apache/opennlp/pull/383#discussion_r535542824



##########
File path: opennlp-tools/src/test/java/opennlp/tools/namefind/NameFinderMETest.java
##########
@@ -109,6 +110,21 @@ public void testNameFinder() throws Exception {
     Assert.assertEquals(new Span(4, 6, DEFAULT), names[1]);
   }
 
+  @Test
+  public void testNameFinderWithDownloadedModel() throws Exception {
+
+    String input = "Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .";
+    String[] sentence = input.split(" ");
+
+    TokenNameFinder nameFinder = new NameFinderME("en", DownloadUtil.EntityType.PERSON);

Review comment:
       I wonder if some of these tests could become flaky if there are internet connection issues in the client or the server side?

##########
File path: opennlp-tools/src/main/java/opennlp/tools/util/DownloadUtil.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+
+import opennlp.tools.chunker.ChunkerModel;
+import opennlp.tools.namefind.TokenNameFinderModel;
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+import opennlp.tools.util.model.BaseModel;
+
+public class DownloadUtil {

Review comment:
       I came looking at this class to check where the models were downloaded from, and to which folder on my computer. Maybe we could add a short Javadoc here, saying the models are downloaded from SourceForge, and are put in the `$USER/.opennlp` folder?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [opennlp] jzonthemtn closed pull request #383: OPENNLP-1318: Add automatic model downloading.

Posted by GitBox <gi...@apache.org>.
jzonthemtn closed pull request #383:
URL: https://github.com/apache/opennlp/pull/383


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [opennlp] jzonthemtn commented on a change in pull request #383: OPENNLP-1318: Add automatic model downloading.

Posted by GitBox <gi...@apache.org>.
jzonthemtn commented on a change in pull request #383:
URL: https://github.com/apache/opennlp/pull/383#discussion_r535605688



##########
File path: opennlp-tools/src/test/java/opennlp/tools/namefind/NameFinderMETest.java
##########
@@ -109,6 +110,21 @@ public void testNameFinder() throws Exception {
     Assert.assertEquals(new Span(4, 6, DEFAULT), names[1]);
   }
 
+  @Test
+  public void testNameFinderWithDownloadedModel() throws Exception {
+
+    String input = "Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .";
+    String[] sentence = input.split(" ");
+
+    TokenNameFinder nameFinder = new NameFinderME("en", DownloadUtil.EntityType.PERSON);

Review comment:
       Updated and move those tests to *IT.java tests outside the regular build.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org