You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opennlp.apache.org by GitBox <gi...@apache.org> on 2022/12/29 18:25:25 UTC
[GitHub] [opennlp] mawiesne opened a new pull request, #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
mawiesne opened a new pull request, #472:
URL: https://github.com/apache/opennlp/pull/472
Change
-
- provides new JUnit test `DownloadUtilTest` to verify `DownloadUtil`
- fixes an incorrect **Dutch** model URL constant (incorrect path element)
- fixes an incorrect **Italian** model URL constant ("sentence" -> "tokens")
- fixes an incorrect **French** model URL constant ("en-ud-ewt2 -> "fr-ud-ftb")
Tasks
-
Thank you for contributing to Apache OpenNLP.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with OPENNLP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn clean install at the root opennlp folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file in opennlp folder?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found in opennlp folder?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] mawiesne commented on pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
mawiesne commented on PR #472:
URL: https://github.com/apache/opennlp/pull/472#issuecomment-1367584475
@rzo1 Introduced `@ EnabledWhenCDNAvailable `. This takes care of the above proposed check for CDN availability. Downloading test cases are now only executed when Apache's CND infrastructure is reachable.
Other improvements -> separate issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] mawiesne commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
mawiesne commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059115709
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
It's actually downloading, like users would require it to download if they create XYZModel instances via the language code constructor.
Atm, I have no "good" idea how to mock the download in the case of the class unter test, that is DownloadUtil. However, I see your idea/point. Maybe worth a separate issue for (further) improvements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] rzo1 commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
rzo1 commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059116766
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
If the files are available at `System.getProperty("user.home") + "/.opennlp/")` and you are offline, the tests will pass. Otherwise if the tests were never run and you are offline, they will fail.
In order to mock it, we would need to adjust the `BASE_URL` part and inject a base url like `file://` pointing to a local location containing related (mock) models. Due to the `static` nature of `DownloadUtil`, this is pain ;-)
Side note: Having hard-coded values is a bit error prone (as shown by this issue ;-)). Perhaps it would be worth to parse the content of https://dlcdn.apache.org/opennlp/models/ud-models-1.0/ and obtain the model urls dynamically from that location (of course via separate issue).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] kinow commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
kinow commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059139171
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
+1 to @rzo1 suggestion to improve it, and also to not block this PR and move it to a follow-up issue. Thanks @mawiesne !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] kinow commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
kinow commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059107085
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
Is the test here actually downloading the model? If so, do you think we could mock it to run offline?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] rzo1 commented on pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
rzo1 commented on PR #472:
URL: https://github.com/apache/opennlp/pull/472#issuecomment-1367882061
> @rzo1 Introduced `@ EnabledWhenCDNAvailable `. This takes care of the above proposed check for CDN availability. Downloading test cases are now only executed when Apache's CND infrastructure is reachable.
>
> Other improvements -> separate issue.
Works fine. I will open an issue regarding the improvement and work on it :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] rzo1 commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
rzo1 commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059116766
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
If the files are available at `System.getProperty("user.home") + "/.opennlp/")` and you are offline, the tests will pass. Otherwise if the tests were never run and you are offline, they will fail.
In order to mock it, we would need to adjust the `BASE_URL` part and inject a base url like `file://` pointing to a local location containing related (mock) models. Due to the `static` nature of `DownloadUtil`, this is pain ;-)
Side note: Having hard-coded values is a bit error prone (as shown by this issue ;-)). Perhaps it would be worth to parse the content of https://dlcdn.apache.org/opennlp/models/ud-models-1.0/ and obtain the model urls dynamically from that location (of course via separate issue). We do something similar in TomEE to dynamically generate our download page for the website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] mawiesne commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
mawiesne commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059121487
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
> If the files are available at System.getProperty("user.home") + "/.opennlp/") and you are offline, the tests will pass. Otherwise if the tests were never run and you are offline, they will fail.
>
I can improve this part with the help of JUnit's `ExecutionCondition`. Stay tuned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] mawiesne commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
mawiesne commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059147295
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
> I can improve this part with the help of JUnit's `ExecutionCondition`. Stay tuned.
Done, pushed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] mawiesne commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
mawiesne commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059115709
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
It's actually downloading, like users would require to download with this util class if they create XYZModel instances via the language code constructor.
Atm, I have no "good" idea how to mock the download in the case of the class unter test, that is DownloadUtil. However, I see your idea/point. Maybe worth a separate issue for (further) improvements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] mawiesne commented on pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
mawiesne commented on PR #472:
URL: https://github.com/apache/opennlp/pull/472#issuecomment-1367590491
> If it's broken there too, then I think it should be fine to fix it too 👍 my 0.02$
>
> https://github.com/apache/opennlp-site/blob/main/src/main/jbake/content/models.ad
@kinow See: https://issues.apache.org/jira/browse/OPENNLP-1427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] jzonthemtn commented on pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
jzonthemtn commented on PR #472:
URL: https://github.com/apache/opennlp/pull/472#issuecomment-1367881339
@mawiesne Thanks for fixing these and the ones on the website. Much appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] mawiesne commented on pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
mawiesne commented on PR #472:
URL: https://github.com/apache/opennlp/pull/472#issuecomment-1367515195
@jzonthemtn FYI: The download page for the models https://opennlp.apache.org/models.html shows the same problem for the "Tokenizer model for French" as I found in this PR. Can you or should I tackle this? wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] jzonthemtn merged pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
jzonthemtn merged PR #472:
URL: https://github.com/apache/opennlp/pull/472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [opennlp] rzo1 commented on a diff in pull request #472: OPENNLP-1414 Investigate why DownloadUtil can't retrieve NL models via a CDN
Posted by GitBox <gi...@apache.org>.
rzo1 commented on code in PR #472:
URL: https://github.com/apache/opennlp/pull/472#discussion_r1059116766
##########
opennlp-tools/src/test/java/opennlp/tools/util/DownloadUtilTest.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.stream.Stream;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.tools.sentdetect.SentenceModel;
+import opennlp.tools.tokenize.TokenizerModel;
+
+public class DownloadUtilTest {
+
+ @ParameterizedTest(name = "Verify \"{0}\" sentence model")
+ @ValueSource(strings = {"en", "fr", "de", "it", "nl"})
+ public void testDownloadModelByLanguage(String lang) throws IOException {
+ SentenceModel model = DownloadUtil.downloadModel(lang,
+ DownloadUtil.ModelType.SENTENCE_DETECTOR, SentenceModel.class);
Review Comment:
If the files are available at `System.getProperty("user.home") + "/.opennlp/")` and you are offline, the tests will pass. Otherwise if the tests were never run and you are offline, they will fail.
In order to mock it, we would need to adjust the `BASE_URL` part and inject a base url like `file://` pointing to a local location containing related (mock) models. Due to the `static` nature of `DownloadUtil`, this is pain ;-)
Side note: Having hard-coded values is a bit error prone (as shown by this issue ;-)). Perhaps it would be worth to parse the content of https://dlcdn.apache.org/opennlp/models/ud-models-1.0/ and obtain the model urls dynamically from that location (of course via separate issue). We do something similar in TomEE to dynamically generate our download page for the website. We do something similar in TomEE to dynamically generate our download page for the website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org