You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Apache Wiki <wi...@apache.org> on 2014/03/05 23:44:20 UTC

[Nutch Wiki] Trivial Update of "IndexStructure" by LewisJohnMcgibbney

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.

The "IndexStructure" page has been changed by LewisJohnMcgibbney:
https://wiki.apache.org/nutch/IndexStructure?action=diff&rev1=18&rev2=19

  
  The index structure formed after indexing is shown below : 
  
- ||'''Field Name'''||'''Stored'''||'''Index'''|| '''Plugin/Class''' ||'''Comment'''||
+ ||'''Field Name'''||'''Stored'''||'''Index'''|| '''Plugin/Class''' ||'''Comment'''|| '''version'''||
+ || || || || || || '''1.x''' || '''2.x''' ||
- || 	boost 	 ||	YES 	|| 	Not Indexed 	|| various scoring plugins || Adds a '''score''' value field to a particular document. This is allocated based upon its importance within the webgraph. ||
+ || 	boost 	 ||	YES 	|| 	Not Indexed 	|| various scoring plugins || Adds a '''score''' value field to a particular document. This is allocated based upon its importance within the webgraph. || ?  || ? ||
- || 	digest 	||	YES 	||	Not Indexed 	|| org.apache.nutch.indexer.IndexerMapReduce.java || Adds a '''message digest''' field to a document. Can be MD5 over content and headers or more sophisticated text profile of the content. ||
+ || 	digest 	||	YES 	||	Not Indexed 	|| org.apache.nutch.indexer.IndexerMapReduce.java || Adds a '''message digest''' field to a document. Can be MD5 over content and headers or more sophisticated text profile of the content. ||  ?  || ? ||
- || 	lang 	||	YES 	||	Un-Tokenized 	||	language-identifier || Add a '''lang''', language field to a document.||
+ || 	lang 	||	YES 	||	Un-Tokenized 	||	language-identifier || Add a '''lang''', language field to a document.||  ?  || ? ||
- || 	segment ||		YES 	||	Not Indexed 	|| org.apache.nutch.indexer.IndexerMapReduce.java || Adds the originating '''segment''' field to the document, used to identify the most recent segment in which this document was fetched. ||
+ || 	segment ||		YES 	||	Not Indexed 	|| org.apache.nutch.indexer.IndexerMapReduce.java || Adds the originating '''segment''' field to the document, used to identify the most recent segment in which this document was fetched. ||  ?  || ? ||
- || 	tstamp	||	YES 	||	Tokenized 	|| /!\ NEEDS COMMENT /!\ || Adds a '''timestamp''' field of the most recent time this document was fetched ||
+ || 	tstamp	||	YES 	||	Tokenized 	|| /!\ NEEDS COMMENT /!\ || Adds a '''timestamp''' field of the most recent time this document was fetched ||  ?  || ? ||
- || 	cc:license	||	YES 	||	Indexed, Tokenized 	|| creativecommons || Adds the entire license as '''cc:license=xxx''' and '''attributes''' extracted of the license url||
+ || 	cc:license	||	YES 	||	Indexed, Tokenized 	|| creativecommons || Adds the entire license as '''cc:license=xxx''' and '''attributes''' extracted of the license url||  ?  || ? ||
- || 	cc:meta	||	YES 	||	Indexed, Tokenized 	||	creativecommons || Adds the license location as '''cc:meta=xxx''' ||
+ || 	cc:meta	||	YES 	||	Indexed, Tokenized 	||	creativecommons || Adds the license location as '''cc:meta=xxx''' ||  ?  || ? ||
- || 	cc:type	||	YES 	||	Indexed,Tokenized 	||	creativecommons || Adds the work type as '''cc:type=xxx'''||
+ || 	cc:type	||	YES 	||	Indexed,Tokenized 	||	creativecommons || Adds the work type as '''cc:type=xxx'''||  ?  || ? ||
- || 	anchor 	||	NO 	||	Tokenized 	||	index-anchor || Indexing filter that indexes all inbound '''anchor text''' for a document.||
+ || 	anchor 	||	NO 	||	Tokenized 	||	index-anchor || Indexing filter that indexes all inbound '''anchor text''' for a document.||  ?  || ? ||
- || 	title 	||	YES	||	Tokenized 	||	index-basic	|| Adds basic searchable '''title field''' to a document. Also indexed by index-more ||
+ || 	title 	||	YES	||	Tokenized 	||	index-basic	|| Adds basic searchable '''title field''' to a document. Also indexed by index-more ||  ?  || ? ||
- || 	site 	||	NO 	||	Un-Tokenized	||	index-basic || Adds basic searchable '''site field''' to a document. ||
+ || 	site 	||	NO 	||	Un-Tokenized	||	index-basic || Adds basic searchable '''site field''' to a document. ||  ?  || ? ||
- || 	host 	||	NO 	||	Tokenized 	||	index-basic 	|| Adds basic searchable '''hostname field''' to a document. ||
+ || 	host 	||	NO 	||	Tokenized 	||	index-basic 	|| Adds basic searchable '''hostname field''' to a document. ||  ?  || ? ||
- || 	url 	||	YES	||	Tokenized 	||	index-basic || Adds basic searchable '''URL field''' to a document. ||
+ || 	url 	||	YES	||	Tokenized 	||	index-basic || Adds basic searchable '''URL field''' to a document. ||  ?  || ? ||
- || 	content 	||	NO 	||	Tokenized	||	index-basic	|| Adds basic searchable '''content field''' to a document. ||
+ || 	content 	||	NO 	||	Tokenized	||	index-basic	|| Adds basic searchable '''content field''' to a document. ||  ?  || ? ||
- || 	lastModified 	||	NO	||	Indexed, Un-Tokenized	||	index-more || Adds some time related meta info in the form of '''last-modified''' if present. ||
+ || 	lastModified 	||	NO	||	Indexed, Un-Tokenized	||	index-more || Adds some time related meta info in the form of '''last-modified''' if present. ||  ?  || ? ||
- || 	date 	||	NO 	||	Indexed, Un-Tokenized	||	index-more || Index date as last-modified, or, if that's not present, uses fetch time. ||
+ || 	date 	||	NO 	||	Indexed, Un-Tokenized	||	index-more || Index date as last-modified, or, if that's not present, uses fetch time. ||  ?  || ? ||
- || 	contentLength	||	NO	||	Indexed, Un-Tokenized 	||	index-more || /!\ NEEDS COMMENT /!\ ||
+ || 	contentLength	||	NO	||	Indexed, Un-Tokenized 	||	index-more || /!\ NEEDS COMMENT /!\ ||  ?  || ? ||
- || 	type 	||	NO 	||	Indexed, Un-Tokenized 	||	index-more 	|| Adds contentType, primaryType, subType (all mime-types) ||
+ || 	type 	||	NO 	||	Indexed, Un-Tokenized 	||	index-more 	|| Adds contentType, primaryType, subType (all mime-types) ||  ?  || ? ||
- || 	primaryType 	||	NO	||	Indexed, Un-Tokenized 	||	index-more	||	primaryType (mime-type) ||
+ || 	primaryType 	||	NO	||	Indexed, Un-Tokenized 	||	index-more	||	primaryType (mime-type) ||  ?  || ? ||
- || 	subType 	||	NO	||	Indexed, Un-Tokenized 	||	index-more 	||	subType (mime-type) ||
+ || 	subType 	||	NO	||	Indexed, Un-Tokenized 	||	index-more 	||	subType (mime-type) ||  ?  || ? ||
- ||      tld             ||     YES      || Un-Tokenized / NotStored(based on conf) || tld || Adds a '''top level domain''' field to the document.  ||
+ ||      tld             ||     YES      || Un-Tokenized / NotStored(based on conf) || tld || Adds a '''top level domain''' field to the document.  ||  ?  || ? ||
- ||      subcollection   ||    YES || Tokenized || subcollection || For Comprehensive description see src/java/org/apache/nutch/collection/'''package.html'''   ||
+ ||      subcollection   ||    YES || Tokenized || subcollection || For Comprehensive description see src/java/org/apache/nutch/collection/'''package.html'''   ||  ?  || ? ||
- || 	urlmeta ||	NO	||	Indexed, Un-Tokenized 	||	urlmeta 	|| Adds any specified '''url metadata tags''' to the document in the index.||
+ || 	urlmeta ||	NO	||	Indexed, Un-Tokenized 	||	urlmeta 	|| Adds any specified '''url metadata tags''' to the document in the index.||  ?  || ? ||
  ----
  Jira Issues about indexing and IndexingFilterPlugins are