You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Stephan Strittmatter (JIRA)" <ji...@apache.org> on 2005/04/01 10:33:23 UTC
[jira] Created: (NUTCH-34) Parsing different content formats
Parsing different content formats
---------------------------------
Key: NUTCH-34
URL: http://issues.apache.org/jira/browse/NUTCH-34
Project: Nutch
Type: Improvement
Components: fetcher
Reporter: Stephan Strittmatter
At the moment Nuch is set up to filter content by config the xml-config file.
There it is also set global how many bytes are loaded.
I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
1. this document type is wanted (because the parser plugin is
installed and activated)
2. how much of the content is required (some plugins need the whole
content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (NUTCH-34) Parsing different content formats
Posted by Jack Tang <hi...@gmail.com>.
Hi Andrzej
For the second question. I don't think it is the "content size limit amount".
In our CMS product, we need to index the content starts from
"<!--Indexware Content Starts Here-->" and ends with "<!--Indexware
Content Ends Here-->". It is easy to change the HtmlParser ....
/Jack
On 4/17/05, Andrzej Bialecki (JIRA) <ji...@apache.org> wrote:
> [ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_62996 ]
>
> Andrzej Bialecki commented on NUTCH-34:
> ----------------------------------------
>
> Currently there is such a "registry", and it is built and maintained by PluginRepository.
>
> So, it seems to me that the only change required here would be to add attributes to each plugin config file (and plugin interface) which inform all plugin users about the following:
>
> * a boolean, whether the plugin can handle incomplete files or not.
>
> * an int, setting the content size limit.
>
> > Parsing different content formats
> > ---------------------------------
> >
> > Key: NUTCH-34
> > URL: http://issues.apache.org/jira/browse/NUTCH-34
> > Project: Nutch
> > Type: Improvement
> > Components: fetcher
> > Reporter: Stephan Strittmatter
> > Priority: Trivial
>
> >
> > At the moment Nuch is set up to filter content by config the xml-config file.
> > There it is also set global how many bytes are loaded.
> > I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> > 1. this document type is wanted (because the parser plugin is
> > installed and activated)
> > 2. how much of the content is required (some plugins need the whole
> > content and some not)
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> If you want more information on JIRA, or have a bug to report see:
> http://www.atlassian.com/software/jira
>
>
[jira] Kommentiert: (NUTCH-34) Parsing different content formats
Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_63038 ]
Stephan Strittmatter commented on NUTCH-34:
-------------------------------------------
Yes I agree, the supprted mime type(s) and file extension(s) should also be "registered" by the parser plugins in order to inform the crowler wether to transfere content or ignore it.
Having this, we could give up this many mails why a specific format is not crawled because it is in the ignore-list of the crawlers config. This config would be obsolete then because the plugins know themselfe what has to be crawled or not.
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (NUTCH-34) Parsing different content formats
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=all ]
Chris A. Mattmann resolved NUTCH-34:
------------------------------------
Fix Version: 0.7.2-dev
0.8-dev
Resolution: Fixed
This issue was addressed via the application of NUTCH-88 applied to Nutch 0.7.2-dev and Nutch 0.8-dev. The only thing not addressed by this issue was the addition of a content-length attribute to a particular parsing plugin declaring a parse plugins ability to handle some content length amount. I recommend if that becomes an important issue to address, that a separate JIRA issue be created for that particular functionality.
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
> Fix For: 0.7.2-dev, 0.8-dev
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Kommentiert: (NUTCH-34) Parsing different content formats
Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_63156 ]
Stephan Strittmatter commented on NUTCH-34:
-------------------------------------------
About the feature to activate/deactivate a plugin, IMHO a plugin should work "automatically" if it is installed to the system. Like within Eclipse for example. But additionaly you should be able to disable an installed plugin as you annotate.
I think in a pluggable architecture it should be as easy as possible to add a new plugin without knowing too much details of the environment. Later, if the user wand to refine the settings, then he has to deal with config files.
I know, th problem is then the "default" ordering. I think, it will be in a way like now.
About the plugin-id, I think the plugins name (plugins.xml) could be taken as ID?
Probably on starting nutch the config file will be generated if it does not exists and the plugins will be stored there. Then you can later modify this file (this is just an idea, not thought to the real end).
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-34) Parsing different content formats
Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_63037 ]
Jerome Charron commented on NUTCH-34:
-------------------------------------
I agree with Stephan, only one int field seems to be needed.
But what about content-type(s) handled by the plugin (adding this feature could avoid to fetch some unecessary content, or at least to store some unecessary content)?
Jerome
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Kommentiert: (NUTCH-34) Parsing different content formats
Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_63033 ]
Stephan Strittmatter commented on NUTCH-34:
-------------------------------------------
Do you think the boolean is really required or should it not be in the same way than in the file-/http-plugins:
The value "0" implies "whole content is required".
What do you think?
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-34) Parsing different content formats
Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=history ]
Stephan Strittmatter updated NUTCH-34:
--------------------------------------
Priority: Trivial (was: Major)
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-34) Parsing different content formats
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_63147 ]
Andrzej Bialecki commented on NUTCH-34:
----------------------------------------
Stephan,
Regarding the urlfilter config file: well, my point was that it would be nice to have a single place to turn on/off various plugins. The alternative is to do it for each plugin separately... We could change the format though - instead of extensions we could perhaps use plugin IDs...? This file could also define the ordering (or priority) of plugins.
Regarding the plugin ordering: parser plugins are somewhat exceptional, because only one of them has to be invoked. Other plugins are used as a filtering chain - but even in those cases their order matters.
For the parsing plugins currently the algorithm works as follows (copied from ParserFactory):
[Parser extensions should define the attributes "contentType" and/or "pathSuffix". Content type has priority: the first plugin found whose "contentType" attribute matches the beginning of the content's type is used. If none match, then the first whose "pathSuffix" attribute matches the end of the url's path is used. If neither of these match, then the first plugin whose "pathSuffix" is the empty string is used.]
This means that if there are more parsers for the same content type and path suffix, only the first on the list will always be used.
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-34) Parsing different content formats
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_63049 ]
Andrzej Bialecki commented on NUTCH-34:
----------------------------------------
Stephan & Jerome,
Let me explain why I think a boolean is useful (though strictly speaking not required, as you noticed). When the Fetcher gets a content header (be it HTTP or other protocol header), it learns about the content type (if present) and the content size. Based on the content type it can select a parse plugin. Now, if the content size exceeds the maximum size set in the plugin, the Fetcher currently has only one choice - to fetch up to maximum size of bytes, pass this partial content to the plugin and pray that it works. However, if we introduce a boolean property with the meaning "plugin can handle partial content", then the Fetcher can make an informed decision whether to fetch the partial content at all. As a result, we can gain significant bandwidth/disk space/CPU savings. Also, this type of information is very easy to provide... Setting the maximum size to "0" has different semantics - it simply means that Fetcher should fetch all content, no matter its size.
Regarding the plugin registry: IMHO it needs a configuration file anyway. There needs to be a mechanism in place to preserve ordering and priority of active plugins (more sophisticated than the current nearly random way), especially if more than one plugin handles the same mime type. I agree that it's convenient if each plugin "registers itself" for handling given mime types, but I would add to that "with certain priority if more than one plugin exists for a given type". Again, IMHO, it is convenient also to have a single place to quickly turn on/off various plugins - it could be a config file, or it could be an API (perhaps both?).
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-34) Parsing different content formats
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_12360147 ]
Chris A. Mattmann commented on NUTCH-34:
----------------------------------------
Hi Folks,
Just wondering: is this issue taken care of by NUTCH-88? It would seem at least some elements of it were (i.e., the single location of parse plugin ordering and such, the ability to have different parse plugins registered to the same mimeType, with priority, etc.). The only thing that isn't really handled by NUTCH-88 is the contentLength addition to the plugin.xml file, etc., but that could (and IMO should) be split into a separate issue.
I recommend closing this issue as the bulk of it was handled by NUTCH-88.
Cheers,
Chris
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Kommentiert: (NUTCH-34) Parsing different content formats
Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_63142 ]
Stephan Strittmatter commented on NUTCH-34:
-------------------------------------------
Andrzej,
about the boolean flag, this is a good objection. You are right it is better to provide this flag also.
Then the fetcher can be limited to a specific file size to be fetched (the parser plugin can not handle partial content) and then files bigger than the defined value are completley ignored.
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (NUTCH-34) Parsing different content formats
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=all ]
Chris A. Mattmann closed NUTCH-34:
----------------------------------
Issue addressed by NUTCH-88.
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
> Fix For: 0.7.2-dev, 0.8-dev
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Kommentiert: (NUTCH-34) Parsing different content formats
Posted by "Stephan Strittmatter (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_63145 ]
Stephan Strittmatter commented on NUTCH-34:
-------------------------------------------
Andrzej,
about the plugin registry:
I also agree with you, it should be possible to order in some way the plugins and activate or deactivate them by config/API. (About API calls Nutch could support then something like selfe "self-healing" if a specific parser fials too often. but this is another story...)
But what I think, the restrictions, which content should be feched and which not by defining the excludes like:
<code>
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|mpg|xls|gz|rpm|tgz|mov|MOV|exe)$
</code>
in crawl-urlfilter.txt should be managed by the parser plugins which are currently active.
BTW, what will happen if there are more than one parser-plugin which feels responsible for a specific content? I haven't tried that. Will this make sense? As fallback if the first parser failed?
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-34) Parsing different content formats
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-34?page=comments#action_62996 ]
Andrzej Bialecki commented on NUTCH-34:
----------------------------------------
Currently there is such a "registry", and it is built and maintained by PluginRepository.
So, it seems to me that the only change required here would be to add attributes to each plugin config file (and plugin interface) which inform all plugin users about the following:
* a boolean, whether the plugin can handle incomplete files or not.
* an int, setting the content size limit.
> Parsing different content formats
> ---------------------------------
>
> Key: NUTCH-34
> URL: http://issues.apache.org/jira/browse/NUTCH-34
> Project: Nutch
> Type: Improvement
> Components: fetcher
> Reporter: Stephan Strittmatter
> Priority: Trivial
>
> At the moment Nuch is set up to filter content by config the xml-config file.
> There it is also set global how many bytes are loaded.
> I think it yould be better to let the parser plugins "register" themselfe in some registry where every plugin could tell the fetcher, that:
> 1. this document type is wanted (because the parser plugin is
> installed and activated)
> 2. how much of the content is required (some plugins need the whole
> content and some not)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira