[GitHub] [opennlp] rzo1 commented on a diff in pull request #473: OPENNLP-1428 - Enhance DownloadUtil to avoid the use of hard-coded model urls

[GitBox <gi...@apache.org>]

rzo1 commented on code in PR #473:
URL: https://github.com/apache/opennlp/pull/473#discussion_r1059494145


##########
opennlp-tools/src/main/java/opennlp/tools/util/DownloadUtil.java:
##########
@@ -174,4 +143,82 @@ public static <T extends BaseModel> T downloadModel(URL url, Class<T> type) thro
     }
   }
 
+  @Internal
+  static class DownloadParser {
+
+    private static final Pattern LINK_PATTERN = Pattern.compile("<a href=\\\"(.*?)\\\">(.*?)</a>", Pattern.CASE_INSENSITIVE | Pattern.DOTALL);
+    private final URL indexUrl;
+
+    DownloadParser(URL indexUrl) {
+      Objects.requireNonNull(indexUrl);
+      this.indexUrl = indexUrl;
+    }
+
+    Map<String, Map<ModelType, String>> getAvailableModels() {
+
+      final Matcher matcher = LINK_PATTERN.matcher(fetchPageIndex());
+
+      final List<String> links = new ArrayList<>();
+      while (matcher.find()) {
+        links.add(matcher.group(1));
+      }
+
+      return toMap(links);
+    }
+
+    private Map<String, Map<ModelType, String>> toMap(List<String> links) {
+
+      final Map<String, Map<ModelType, String>> result = new HashMap<>();
+
+      for (String link : links) {

Review Comment:
   We are voting on models [1], so perhaps it would make sense to add the structure of the model file names to [2].
   
   I think, that the release process for models is slightly different, so we maybe need a separate page for "Making a model release" with instructions for conducting it. 
   
   However, the structure itself is indeed defined and described, see [3]. It is a bit hidden though.
   
   
   [1] https://lists.apache.org/thread/t2v946x3s24yvvfc6hpq41lqnxshx2b0
   [2] https://opennlp.apache.org/release.html
   [3] https://dist.apache.org/repos/dist/dev/opennlp/ud-models-1.0/README



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org