You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Apache Wiki <wi...@apache.org> on 2014/03/05 23:44:20 UTC
[Nutch Wiki] Trivial Update of "IndexStructure" by LewisJohnMcgibbney
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The "IndexStructure" page has been changed by LewisJohnMcgibbney:
https://wiki.apache.org/nutch/IndexStructure?action=diff&rev1=18&rev2=19
The index structure formed after indexing is shown below :
- ||'''Field Name'''||'''Stored'''||'''Index'''|| '''Plugin/Class''' ||'''Comment'''||
+ ||'''Field Name'''||'''Stored'''||'''Index'''|| '''Plugin/Class''' ||'''Comment'''|| '''version'''||
+ || || || || || || '''1.x''' || '''2.x''' ||
- || boost || YES || Not Indexed || various scoring plugins || Adds a '''score''' value field to a particular document. This is allocated based upon its importance within the webgraph. ||
+ || boost || YES || Not Indexed || various scoring plugins || Adds a '''score''' value field to a particular document. This is allocated based upon its importance within the webgraph. || ? || ? ||
- || digest || YES || Not Indexed || org.apache.nutch.indexer.IndexerMapReduce.java || Adds a '''message digest''' field to a document. Can be MD5 over content and headers or more sophisticated text profile of the content. ||
+ || digest || YES || Not Indexed || org.apache.nutch.indexer.IndexerMapReduce.java || Adds a '''message digest''' field to a document. Can be MD5 over content and headers or more sophisticated text profile of the content. || ? || ? ||
- || lang || YES || Un-Tokenized || language-identifier || Add a '''lang''', language field to a document.||
+ || lang || YES || Un-Tokenized || language-identifier || Add a '''lang''', language field to a document.|| ? || ? ||
- || segment || YES || Not Indexed || org.apache.nutch.indexer.IndexerMapReduce.java || Adds the originating '''segment''' field to the document, used to identify the most recent segment in which this document was fetched. ||
+ || segment || YES || Not Indexed || org.apache.nutch.indexer.IndexerMapReduce.java || Adds the originating '''segment''' field to the document, used to identify the most recent segment in which this document was fetched. || ? || ? ||
- || tstamp || YES || Tokenized || /!\ NEEDS COMMENT /!\ || Adds a '''timestamp''' field of the most recent time this document was fetched ||
+ || tstamp || YES || Tokenized || /!\ NEEDS COMMENT /!\ || Adds a '''timestamp''' field of the most recent time this document was fetched || ? || ? ||
- || cc:license || YES || Indexed, Tokenized || creativecommons || Adds the entire license as '''cc:license=xxx''' and '''attributes''' extracted of the license url||
+ || cc:license || YES || Indexed, Tokenized || creativecommons || Adds the entire license as '''cc:license=xxx''' and '''attributes''' extracted of the license url|| ? || ? ||
- || cc:meta || YES || Indexed, Tokenized || creativecommons || Adds the license location as '''cc:meta=xxx''' ||
+ || cc:meta || YES || Indexed, Tokenized || creativecommons || Adds the license location as '''cc:meta=xxx''' || ? || ? ||
- || cc:type || YES || Indexed,Tokenized || creativecommons || Adds the work type as '''cc:type=xxx'''||
+ || cc:type || YES || Indexed,Tokenized || creativecommons || Adds the work type as '''cc:type=xxx'''|| ? || ? ||
- || anchor || NO || Tokenized || index-anchor || Indexing filter that indexes all inbound '''anchor text''' for a document.||
+ || anchor || NO || Tokenized || index-anchor || Indexing filter that indexes all inbound '''anchor text''' for a document.|| ? || ? ||
- || title || YES || Tokenized || index-basic || Adds basic searchable '''title field''' to a document. Also indexed by index-more ||
+ || title || YES || Tokenized || index-basic || Adds basic searchable '''title field''' to a document. Also indexed by index-more || ? || ? ||
- || site || NO || Un-Tokenized || index-basic || Adds basic searchable '''site field''' to a document. ||
+ || site || NO || Un-Tokenized || index-basic || Adds basic searchable '''site field''' to a document. || ? || ? ||
- || host || NO || Tokenized || index-basic || Adds basic searchable '''hostname field''' to a document. ||
+ || host || NO || Tokenized || index-basic || Adds basic searchable '''hostname field''' to a document. || ? || ? ||
- || url || YES || Tokenized || index-basic || Adds basic searchable '''URL field''' to a document. ||
+ || url || YES || Tokenized || index-basic || Adds basic searchable '''URL field''' to a document. || ? || ? ||
- || content || NO || Tokenized || index-basic || Adds basic searchable '''content field''' to a document. ||
+ || content || NO || Tokenized || index-basic || Adds basic searchable '''content field''' to a document. || ? || ? ||
- || lastModified || NO || Indexed, Un-Tokenized || index-more || Adds some time related meta info in the form of '''last-modified''' if present. ||
+ || lastModified || NO || Indexed, Un-Tokenized || index-more || Adds some time related meta info in the form of '''last-modified''' if present. || ? || ? ||
- || date || NO || Indexed, Un-Tokenized || index-more || Index date as last-modified, or, if that's not present, uses fetch time. ||
+ || date || NO || Indexed, Un-Tokenized || index-more || Index date as last-modified, or, if that's not present, uses fetch time. || ? || ? ||
- || contentLength || NO || Indexed, Un-Tokenized || index-more || /!\ NEEDS COMMENT /!\ ||
+ || contentLength || NO || Indexed, Un-Tokenized || index-more || /!\ NEEDS COMMENT /!\ || ? || ? ||
- || type || NO || Indexed, Un-Tokenized || index-more || Adds contentType, primaryType, subType (all mime-types) ||
+ || type || NO || Indexed, Un-Tokenized || index-more || Adds contentType, primaryType, subType (all mime-types) || ? || ? ||
- || primaryType || NO || Indexed, Un-Tokenized || index-more || primaryType (mime-type) ||
+ || primaryType || NO || Indexed, Un-Tokenized || index-more || primaryType (mime-type) || ? || ? ||
- || subType || NO || Indexed, Un-Tokenized || index-more || subType (mime-type) ||
+ || subType || NO || Indexed, Un-Tokenized || index-more || subType (mime-type) || ? || ? ||
- || tld || YES || Un-Tokenized / NotStored(based on conf) || tld || Adds a '''top level domain''' field to the document. ||
+ || tld || YES || Un-Tokenized / NotStored(based on conf) || tld || Adds a '''top level domain''' field to the document. || ? || ? ||
- || subcollection || YES || Tokenized || subcollection || For Comprehensive description see src/java/org/apache/nutch/collection/'''package.html''' ||
+ || subcollection || YES || Tokenized || subcollection || For Comprehensive description see src/java/org/apache/nutch/collection/'''package.html''' || ? || ? ||
- || urlmeta || NO || Indexed, Un-Tokenized || urlmeta || Adds any specified '''url metadata tags''' to the document in the index.||
+ || urlmeta || NO || Indexed, Un-Tokenized || urlmeta || Adds any specified '''url metadata tags''' to the document in the index.|| ? || ? ||
----
Jira Issues about indexing and IndexingFilterPlugins are