You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@oodt.apache.org by Sheryl John <sh...@gmail.com> on 2011/03/19 08:22:02 UTC

Precondition Comparator in Crawler Framework

Hi,


Under the Comparator package in the OODT CAS Crawler Framework, there is one
class FilemgrUniqunessCheckComparator( with method performCheck) and though
this has to be satisfied before metadata extraction and ingestion, I don't
see any class dependent on this class or calling the method performCheck.

However, under the package there is another class FilemgrUniquenessChecker
whose method performAction is called in CrawlerAction.

Have I missed out any class where the FilemgrUniqunessCheckComparator is
dependent on?

-Sheryl

Re: Precondition Comparator in Crawler Framework

Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.

Hi Sheryl,

> 
> Under the Comparator package in the OODT CAS Crawler Framework, there is one class FilemgrUniqunessCheckComparator( with method performCheck) and though this has to be satisfied before metadata extraction and ingestion, I don't see any class dependent on this class or calling the method performCheck.

I'm assuming yo mean this class:

http://svn.apache.org/repos/asf/oodt/trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/comparator/FilemgrUniquenessCheckComparator.java

> 
> However, under the package there is another class FilemgrUniquenessChecker whose method performAction is called in CrawlerAction.  

I'm assuming you mean this class:

> http://svn.apache.org/repos/asf/oodt/trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/action/FilemgrUniquenessChecker.java

> Have I missed out any class where the FilemgrUniqunessCheckComparator is dependent on? 

Nope there is just a small subtlety here:

1. Crawler PreconditionComparators are used to either allow, or prevent metadata to be generated
2. Crawler Actions attach behavior to the preIngestion phase, the postIngestionSuccess phase, or the postingestionFailure phase of the crawler's state model *after* metadata has been generated.

So in the case of your first example, it's a Filemgr Check that *prevents metadata from being generated * (and thus ingestion), if the product has already been cataloged in an FM. In the case of your second example, it's a Filemgr Check *after metadata has been generated* *but before ingestion* to attach to the crawler phase model to prevent ingestion if you have already ingested that product into the FM.

In both cases the classes are called via Spring, when the crawler is running via attaching actions and or comparators to the crawler state model or its pre condition identifiers.

HTH,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Re: Precondition Comparator in Crawler Framework

Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.

Hi Sheryl,

> 
> Under the Comparator package in the OODT CAS Crawler Framework, there is one class FilemgrUniqunessCheckComparator( with method performCheck) and though this has to be satisfied before metadata extraction and ingestion, I don't see any class dependent on this class or calling the method performCheck.

I'm assuming yo mean this class:

http://svn.apache.org/repos/asf/oodt/trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/comparator/FilemgrUniquenessCheckComparator.java

> 
> However, under the package there is another class FilemgrUniquenessChecker whose method performAction is called in CrawlerAction.  

I'm assuming you mean this class:

> http://svn.apache.org/repos/asf/oodt/trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/action/FilemgrUniquenessChecker.java

> Have I missed out any class where the FilemgrUniqunessCheckComparator is dependent on? 

Nope there is just a small subtlety here:

1. Crawler PreconditionComparators are used to either allow, or prevent metadata to be generated
2. Crawler Actions attach behavior to the preIngestion phase, the postIngestionSuccess phase, or the postingestionFailure phase of the crawler's state model *after* metadata has been generated.

So in the case of your first example, it's a Filemgr Check that *prevents metadata from being generated * (and thus ingestion), if the product has already been cataloged in an FM. In the case of your second example, it's a Filemgr Check *after metadata has been generated* *but before ingestion* to attach to the crawler phase model to prevent ingestion if you have already ingested that product into the FM.

In both cases the classes are called via Spring, when the crawler is running via attaching actions and or comparators to the crawler state model or its pre condition identifiers.

HTH,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++