You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "tcurran@approachingpi.com (JIRA)" <ji...@apache.org> on 2009/11/10 02:25:32 UTC
[jira] Created: (NUTCH-764) Add support for vfsfile:// loading of
plugins for JBoss
Add support for vfsfile:// loading of plugins for JBoss
-------------------------------------------------------
Key: NUTCH-764
URL: https://issues.apache.org/jira/browse/NUTCH-764
Project: Nutch
Issue Type: Improvement
Components: searcher
Affects Versions: 1.0.0
Environment: JBoss AS 5.1.0
Reporter: tcurran@approachingpi.com
Priority: Trivial
In the file:
/src/java/org/apache/nutch/plugin/PluginManifestParser.java
There is a check to make sure that the plugin file location is a url formatted like "file://path/plugins".
When deployed on Jboss, the file protocol will sometimes be: "vfsfile://path/plugins". The code with vfsfile can operate the same so I propose a change to the check to also allow this protocol. This would allow Nutch to be deployed on the newer versions of JBoss without any modification.
The altered block of code would look like this:
if (url == null && directory.exists() && directory.isDirectory()
&& directory.listFiles().length > 0) {
return directory; // relative path that is not in the classpath
} else if (url == null) {
LOG.warn("Plugins: directory not found: " + name);
return null;
} else if (!"file".equals(url.getProtocol()) &&
!"vfsfile".equals(url.getProtocol())) {
LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
return null;
}
Index: src/java/org/apache/nutch/plugin/PluginManifestParser.java
===================================================================
--- src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
+++ src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
@@ -121,7 +121,8 @@
} else if (url == null) {
LOG.warn("Plugins: directory not found: " + name);
return null;
- } else if (!"file".equals(url.getProtocol())) {
+ } else if (!"file".equals(url.getProtocol()) &&
+ !"vfsfile".equals(url.getProtocol())) {
LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
return null;
}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-764) Add support for vfsfile:// loading of
plugins for JBoss
Posted by "tcurran@approachingpi.com (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tcurran@approachingpi.com updated NUTCH-764:
--------------------------------------------
Description:
In the file:
/src/java/org/apache/nutch/plugin/PluginManifestParser.java
There is a check to make sure that the plugin file location is a url formatted like "file://path/plugins".
When deployed on Jboss, the file protocol will sometimes be: "vfsfile://path/plugins". The code with vfsfile can operate the same so I propose a change to the check to also allow this protocol. This would allow Nutch to be deployed on the newer versions of JBoss without any modification.
Here is a simple patch:
Index: src/java/org/apache/nutch/plugin/PluginManifestParser.java
===================================================================
--- src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
+++ src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
@@ -121,7 +121,8 @@
} else if (url == null) {
LOG.warn("Plugins: directory not found: " + name);
return null;
- } else if (!"file".equals(url.getProtocol())) {
+ } else if (!"file".equals(url.getProtocol()) &&
+ !"vfsfile".equals(url.getProtocol())) {
LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
return null;
}
was:
In the file:
/src/java/org/apache/nutch/plugin/PluginManifestParser.java
There is a check to make sure that the plugin file location is a url formatted like "file://path/plugins".
When deployed on Jboss, the file protocol will sometimes be: "vfsfile://path/plugins". The code with vfsfile can operate the same so I propose a change to the check to also allow this protocol. This would allow Nutch to be deployed on the newer versions of JBoss without any modification.
The altered block of code would look like this:
if (url == null && directory.exists() && directory.isDirectory()
&& directory.listFiles().length > 0) {
return directory; // relative path that is not in the classpath
} else if (url == null) {
LOG.warn("Plugins: directory not found: " + name);
return null;
} else if (!"file".equals(url.getProtocol()) &&
!"vfsfile".equals(url.getProtocol())) {
LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
return null;
}
Index: src/java/org/apache/nutch/plugin/PluginManifestParser.java
===================================================================
--- src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
+++ src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
@@ -121,7 +121,8 @@
} else if (url == null) {
LOG.warn("Plugins: directory not found: " + name);
return null;
- } else if (!"file".equals(url.getProtocol())) {
+ } else if (!"file".equals(url.getProtocol()) &&
+ !"vfsfile".equals(url.getProtocol())) {
LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
return null;
}
> Add support for vfsfile:// loading of plugins for JBoss
> -------------------------------------------------------
>
> Key: NUTCH-764
> URL: https://issues.apache.org/jira/browse/NUTCH-764
> Project: Nutch
> Issue Type: Improvement
> Components: searcher
> Affects Versions: 1.0.0
> Environment: JBoss AS 5.1.0
> Reporter: tcurran@approachingpi.com
> Priority: Trivial
>
> In the file:
> /src/java/org/apache/nutch/plugin/PluginManifestParser.java
> There is a check to make sure that the plugin file location is a url formatted like "file://path/plugins".
> When deployed on Jboss, the file protocol will sometimes be: "vfsfile://path/plugins". The code with vfsfile can operate the same so I propose a change to the check to also allow this protocol. This would allow Nutch to be deployed on the newer versions of JBoss without any modification.
> Here is a simple patch:
> Index: src/java/org/apache/nutch/plugin/PluginManifestParser.java
> ===================================================================
> --- src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
> +++ src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
> @@ -121,7 +121,8 @@
> } else if (url == null) {
> LOG.warn("Plugins: directory not found: " + name);
> return null;
> - } else if (!"file".equals(url.getProtocol())) {
> + } else if (!"file".equals(url.getProtocol()) &&
> + !"vfsfile".equals(url.getProtocol())) {
> LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
> return null;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-764) Add support for vfsfile:// loading of
plugins for JBoss
Posted by "tcurran@approachingpi.com (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775434#action_12775434 ]
tcurran@approachingpi.com commented on NUTCH-764:
-------------------------------------------------
I should have said, by default, it will always be vfsfile. Jboss 5.0+ uses this virtual file system for any files access performed via the context class loaders. For files outside the deployment directory can still be loaded with file:. It is just a cache and abstracts that sometimes the file will be in memory or a different physical location during application runtime.
for your second question: yes, this does fix Nutch and it is then able to load all of the plugins properly.
I understand your concerns and it was hasty of me to assume that this was a trivial issue to implement. I will spend some time investigating and see if there are any other aspects of Nutch that are impacted by this protocol.
> Add support for vfsfile:// loading of plugins for JBoss
> -------------------------------------------------------
>
> Key: NUTCH-764
> URL: https://issues.apache.org/jira/browse/NUTCH-764
> Project: Nutch
> Issue Type: Improvement
> Components: searcher
> Affects Versions: 1.0.0
> Environment: JBoss AS 5.1.0
> Reporter: tcurran@approachingpi.com
> Priority: Trivial
>
> In the file:
> /src/java/org/apache/nutch/plugin/PluginManifestParser.java
> There is a check to make sure that the plugin file location is a url formatted like "file://path/plugins".
> When deployed on Jboss, the file protocol will sometimes be: "vfsfile://path/plugins". The code with vfsfile can operate the same so I propose a change to the check to also allow this protocol. This would allow Nutch to be deployed on the newer versions of JBoss without any modification.
> Here is a simple patch:
> Index: src/java/org/apache/nutch/plugin/PluginManifestParser.java
> ===================================================================
> --- src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
> +++ src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
> @@ -121,7 +121,8 @@
> } else if (url == null) {
> LOG.warn("Plugins: directory not found: " + name);
> return null;
> - } else if (!"file".equals(url.getProtocol())) {
> + } else if (!"file".equals(url.getProtocol()) &&
> + !"vfsfile".equals(url.getProtocol())) {
> LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
> return null;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-764) Add support for vfsfile:// loading of
plugins for JBoss
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775365#action_12775365 ]
Andrzej Bialecki commented on NUTCH-764:
-----------------------------------------
First question is: why is it sometimes vfsfile:// ? What you propose is just hiding the problem, and not solving it - Nutch does not support this protocol, so we should not pretend to do so. More specifically, plugins can only be loaded from locations accessible through the file:// protocol, and pretending that we can load them from vfsfile:// (whatever it may be) is just not true, even if it works in your specific case.
Second question: does it work with this fix, i.e. is Nutch able to load the plugins using this protocol?
> Add support for vfsfile:// loading of plugins for JBoss
> -------------------------------------------------------
>
> Key: NUTCH-764
> URL: https://issues.apache.org/jira/browse/NUTCH-764
> Project: Nutch
> Issue Type: Improvement
> Components: searcher
> Affects Versions: 1.0.0
> Environment: JBoss AS 5.1.0
> Reporter: tcurran@approachingpi.com
> Priority: Trivial
>
> In the file:
> /src/java/org/apache/nutch/plugin/PluginManifestParser.java
> There is a check to make sure that the plugin file location is a url formatted like "file://path/plugins".
> When deployed on Jboss, the file protocol will sometimes be: "vfsfile://path/plugins". The code with vfsfile can operate the same so I propose a change to the check to also allow this protocol. This would allow Nutch to be deployed on the newer versions of JBoss without any modification.
> Here is a simple patch:
> Index: src/java/org/apache/nutch/plugin/PluginManifestParser.java
> ===================================================================
> --- src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
> +++ src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
> @@ -121,7 +121,8 @@
> } else if (url == null) {
> LOG.warn("Plugins: directory not found: " + name);
> return null;
> - } else if (!"file".equals(url.getProtocol())) {
> + } else if (!"file".equals(url.getProtocol()) &&
> + !"vfsfile".equals(url.getProtocol())) {
> LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
> return null;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-764) Add support for vfsfile:// loading of
plugins for JBoss
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776064#action_12776064 ]
Andrzej Bialecki commented on NUTCH-764:
-----------------------------------------
Thanks for the explanation. Well, it's surprising that it works - in that method in the manifest parser we return a java.io.File constructed from this URL, and we expect that it's a regular java.io.File directory. Next question is: if this is a cache, then perhaps the first time you obtain a resource it returns a file:// but on subsequent calls it returns vfsfile:// and you didn't experience any problems because plugins were loaded just once? I'm guessing here - I don't use JBoss.
> Add support for vfsfile:// loading of plugins for JBoss
> -------------------------------------------------------
>
> Key: NUTCH-764
> URL: https://issues.apache.org/jira/browse/NUTCH-764
> Project: Nutch
> Issue Type: Improvement
> Components: searcher
> Affects Versions: 1.0.0
> Environment: JBoss AS 5.1.0
> Reporter: tcurran@approachingpi.com
> Priority: Trivial
>
> In the file:
> /src/java/org/apache/nutch/plugin/PluginManifestParser.java
> There is a check to make sure that the plugin file location is a url formatted like "file://path/plugins".
> When deployed on Jboss, the file protocol will sometimes be: "vfsfile://path/plugins". The code with vfsfile can operate the same so I propose a change to the check to also allow this protocol. This would allow Nutch to be deployed on the newer versions of JBoss without any modification.
> Here is a simple patch:
> Index: src/java/org/apache/nutch/plugin/PluginManifestParser.java
> ===================================================================
> --- src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
> +++ src/java/org/apache/nutch/plugin/PluginManifestParser.java Mon Nov 09 20:20:51 EST 2009
> @@ -121,7 +121,8 @@
> } else if (url == null) {
> LOG.warn("Plugins: directory not found: " + name);
> return null;
> - } else if (!"file".equals(url.getProtocol())) {
> + } else if (!"file".equals(url.getProtocol()) &&
> + !"vfsfile".equals(url.getProtocol())) {
> LOG.warn("Plugins: not a file: url. Can't load plugins from: " + url);
> return null;
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.