svn commit: r425872 - /maven/repository-manager/trunk/maven-repository-indexer/src/site/apt/design.apt

[br...@apache.org]

Author: brett
Date: Wed Jul 26 15:06:23 2006
New Revision: 425872

URL: http://svn.apache.org/viewvc?rev=425872&view=rev
Log:
[MRM-127] update intended design

Modified:
    maven/repository-manager/trunk/maven-repository-indexer/src/site/apt/design.apt

Modified: maven/repository-manager/trunk/maven-repository-indexer/src/site/apt/design.apt
URL: http://svn.apache.org/viewvc/maven/repository-manager/trunk/maven-repository-indexer/src/site/apt/design.apt?rev=425872&r1=425871&r2=425872&view=diff
==============================================================================
--- maven/repository-manager/trunk/maven-repository-indexer/src/site/apt/design.apt (original)
+++ maven/repository-manager/trunk/maven-repository-indexer/src/site/apt/design.apt Wed Jul 26 15:06:23 2006
@@ -11,16 +11,18 @@
   <<Note: The current indexer design is under review. This document will grow into what it should be, and the code and
   tests refactored to match>>
 
+  ~~TODO: separate API design from Lucene implementation design
+
 * Standard Artifact Index
 
   We currently want to index these elements from the repository:
 
-    * for each artifact file: the artifact ID, version, group ID, classifier, type (extension), filename,
-      checksums (md5, sha1) and size
+    * for each artifact file: the artifact ID, version, group ID, classifier, type (extension), filename (including path
+      from the repository base), checksums (md5, sha1) and size
 
     * for each artifact POM: the packaging, licenses, dependencies, build plugins, reporting plugins
 
-    * plugin prefix from the repository metadata (in the future, more may be indexed)
+    * plugin prefix
 
     * Java classes and packages within a JAR artifact (delimited by \n)
 
@@ -32,23 +34,42 @@
   record may need to be updated when different files that are related to the same artifact are discovered (ie, the
   POM, or for plugins the metadata that contains their prefix).
 
-  Records in the index are generally keyed by their dependency conflict ID (ie, a combination of group, artifact,
-  version, type  and classifier). The exception to this rule is the POM: if an entry already exists with a different
-  type but the same group, artifact, version and no classifier, then a POM entry is not added and the model fields are
-  applied to the existing entry. Conversely, if a POM is added first and an artifact with the same group, artifact,
-  version and no classifier is later added then it overwrites the record of the POM.
-
-  The above process, especially with regard to the handling of the POM, should be much simpler if the discoverer is
-  able to associate a POM to the artifact instead of feeding them in separately as it does at present.
-
-  While some of the information stored is specific to a particular type of file, it is all maintained in a single index
-  for simplicity. In the future, if the content of the various documents diverges greatly, it may be split into separate
-  indexes. In that case, we may consider using Lucene's
-  {{{http://wiki.apache.org/jakarta-lucene/LuceneFAQ#head-b11296f9e7b2a5e7496d67118d0a5898f2fd9823} multiple index
-  searching capabilities}}.
+  To simplify this, the process for discovery is as follows:
 
-  Currently, the discoverer returns POMs as separate artifact entries to the actual artifact, and any derived artifacts
-  in the repository. To accommodate this, when indexed
+    * Discovered artifacts will read the related POM and metadata from the repository to index, rather than relying on
+      it being discovered. This ensures that partial discovery still yields correct results in all cases, and it is
+      possible to construct the entire record without having to read back from the index.
+
+    * POMs that do not have a packaging of POM are not sent to the indexer.
+
+  The result of this process is that updates to a POM or repository metadata and not the corresponding artifact(s) will
+  not update the index. As POMs should not be modified, this will not be a major concern. Likewise, updates to metadata
+  will only accompany updates to the artifact itself, so will not cause a problem.
+
+  The above case may have a problem if the discovery happens during the middle of a deployment outside of the
+  repository manager (where the artifact is present, but the metadata or POM is not). To avoid such cases, the
+  discoverer should only detect changes more than a minute old (this blackout should be configurable).
+
+  Other techniques were considered:
+
+    * Processing each artifact file individually, updating each record as needed.  This would result in having to read
+      back each index record before writing. This is quite costly in Lucene as it would be "read, delete, add". You
+      must have a reader and writer open for that process, and it greatly complicates the code.
+
+    * Have three indices, one for each. This would complicate searching (and may affect ranking of results, though this
+      was not analysed). While Lucene is
+      {{{http://wiki.apache.org/jakarta-lucene/LuceneFAQ#head-b11296f9e7b2a5e7496d67118d0a5898f2fd9823} capable of
+      searching multiple indices}}, it is expected that the results would be in the form of a list of separate records
+      rather than the "table join" this effectively is. A similar derivative of this technique would be to store
+      everything in one index, using a field (previously, doctype) to identify each record.
+
+  Records in the index are keyed by their path from the repository root. While this is longer than using the
+  dependency conflict ID, Lucene cannot delete by a combination of terms, so would require storing an additional
+  field in the index where the file already exists.
+
+  The plugin prefix can be found either from inside the plugin JAR (<<<META-INF/maven/plugin.xml>>>), or from the
+  repository metadata for the plugin's group. For simplicity, the first approach will be used. This means at present
+  there is no need to index the repository metadata, however that may be considered in future.
 
   Note that archetypes currently don't have a packaging associated with them in Maven, so it is not recorded in the POM.
   However, to be able to search by this type, the indexer will look for a <<<META-INF/maven/archetype.xml>>> file, and
@@ -83,7 +104,7 @@
 
     * <<<m>>>: md5 checksum of the JAR
 
-  Only JARs are indexed at present.
+  Only JARs are indexed at present. The JAR filename is used as the key for later deleting entries.
 
 * Searching
 
@@ -92,9 +113,13 @@
 
   Some features that will be available:
 
-    * <Search by a particular field (exact match)>: This would be needed for search by checksum
+    * <Search through most fields for a particular keyword>: the general case described above.
+
+    * <Search by a particular field (exact match)>: This would be needed for search by checksum.
 
-    * <Search in a range of field values>: This would be needed for searching based on update time
+    * <Search in a range of field values>: This would be needed for searching based on update time. Note that in
+      Lucene it may be better to search by other fields (or return all), and then filter the results by dates rather
+      than making dates part of a search query.
 
     * <Limit search to particular fields>: It will be useful to only search Java classes and packages, for example
 
@@ -102,10 +127,3 @@
   reasons. It should not have to read any metadata files or properties of files such as size and checksum from the disk.
   This enables searching a repository remotely without having the physical repository available, which is useful for
   IDE integration among other things.
-
-* Limitations
-
-  Currently, because the POM and artifacts are fed in separately, there is no way to associate an artifact with a
-  classifier to its POM, meaning there is less information about it in the index. It may be best that this occurs by
-  design - it seems that while it is desirable to search by classifier you only want to find the main artifact for
-  browsing and see the derived artifact listed under that. How this evolves should be carefully considered.