You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org> on 2012/10/31 17:15:12 UTC
[jira] [Created] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Rogério Pereira Araújo created NUTCH-1483:
---------------------------------------------
Summary: Can't crawl filesystem with protocol-file plugin
Key: NUTCH-1483
URL: https://issues.apache.org/jira/browse/NUTCH-1483
Project: Nutch
Issue Type: Bug
Components: protocol
Affects Versions: 2.1
Environment: OpenSUSE 12.1, OpenJDK 1.6.0
Reporter: Rogério Pereira Araújo
I tried to follow the same steps describe in this wiki page:
http://wiki.apache.org/nutch/IntranetDocumentSearch
I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
file:///home/rogerio/Documents/
The permissions are ok, I'm running nutch in the same use as folder, so nutch has the required permissions, unfortunatelly I'm getting the following error:
org.apache.nutch.protocol.file.FileError: File Error: 404
at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
Note: The regex-urlfilter entry only works as expected if I add the entry
+^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rogério Pereira Araújo updated NUTCH-1483:
------------------------------------------
Description:
I tried to follow the same steps described in this wiki page:
http://wiki.apache.org/nutch/IntranetDocumentSearch
I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
file:///home/rogerio/Documents/
The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
org.apache.nutch.protocol.file.FileError: File Error: 404
at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
Note: The regex-urlfilter entry only works as expected if I add the entry
+^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
was:
I tried to follow the same steps described in this wiki page:
http://wiki.apache.org/nutch/IntranetDocumentSearch
I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
file:///home/rogerio/Documents/
The permissions are ok, I'm running nutch in the same use as folder, so nutch has the required permissions, unfortunatelly I'm getting the following error:
org.apache.nutch.protocol.file.FileError: File Error: 404
at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
Note: The regex-urlfilter entry only works as expected if I add the entry
+^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0
> Reporter: Rogério Pereira Araújo
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rogério Pereira Araújo updated NUTCH-1483:
------------------------------------------
Description:
I tried to follow the same steps described in this wiki page:
http://wiki.apache.org/nutch/IntranetDocumentSearch
I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
file:///home/rogerio/Documents/
The permissions are ok, I'm running nutch in the same use as folder, so nutch has the required permissions, unfortunatelly I'm getting the following error:
org.apache.nutch.protocol.file.FileError: File Error: 404
at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
Note: The regex-urlfilter entry only works as expected if I add the entry
+^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
was:
I tried to follow the same steps describe in this wiki page:
http://wiki.apache.org/nutch/IntranetDocumentSearch
I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
file:///home/rogerio/Documents/
The permissions are ok, I'm running nutch in the same use as folder, so nutch has the required permissions, unfortunatelly I'm getting the following error:
org.apache.nutch.protocol.file.FileError: File Error: 404
at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
Note: The regex-urlfilter entry only works as expected if I add the entry
+^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0
> Reporter: Rogério Pereira Araújo
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch in the same use as folder, so nutch has the required permissions, unfortunatelly I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488390#comment-13488390 ]
Rogério Pereira Araújo commented on NUTCH-1483:
-----------------------------------------------
Patch applied and the workaround that you proposed resolved my issue, now I can crawl the filesystem.
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
> Attachments: NUTCH-1483.patch
>
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Sebastian Nagel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488558#comment-13488558 ]
Sebastian Nagel commented on NUTCH-1483:
----------------------------------------
Thanks!
Issue with un-reversing URLs pulled out to NUTCH-1484 since it's more critical (no work-around).
Fixing the URL normalizers (and filters, see last comment) will take more time.
Btw., {{file://localhost/Documents/}} is the only legal for according to [RFC 1738|http://tools.ietf.org/html/rfc1738] (1994) while {{file:///Documents/}} is allowed by [RFC 3986|http://tools.ietf.org/html/rfc3986]:
{quote}
the "file" URI scheme is defined so that no authority, an empty host, and "localhost" all mean the end-user's machine
{quote}
Maybe we could also make protocol-file more lazy.
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
> Attachments: NUTCH-1483.patch
>
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Sebastian Nagel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebastian Nagel updated NUTCH-1483:
-----------------------------------
Attachment: NUTCH-1483.patch
StringUtils.split(String, char) does not preserve empty parts: host is empty in case of file: URLs.
Patch includes a test case.
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
> Attachments: NUTCH-1483.patch
>
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488397#comment-13488397 ]
Rogério Pereira Araújo commented on NUTCH-1483:
-----------------------------------------------
One important thing to mention, if I add the following regex to regex-urlfilter:
+^file:///home/rogerio/Documents
No documents gets crawled, in order to get documents crawled I must leave the default which is:
+^.
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
> Attachments: NUTCH-1483.patch
>
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Sebastian Nagel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488200#comment-13488200 ]
Sebastian Nagel commented on NUTCH-1483:
----------------------------------------
I tried with 1.x/trunk.
For 2.x URLs with only one slash breaks usage of reverted URLs.
Have you tried removing the regex normalizer rule?
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0
> Reporter: Rogério Pereira Araújo
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Sebastian Nagel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488254#comment-13488254 ]
Sebastian Nagel commented on NUTCH-1483:
----------------------------------------
Rogério, can you apply the patch, re-compile and try again?
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
> Attachments: NUTCH-1483.patch
>
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rogério Pereira Araújo updated NUTCH-1483:
------------------------------------------
Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4 (was: OpenSUSE 12.1, OpenJDK 1.6.0)
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Sebastian Nagel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488558#comment-13488558 ]
Sebastian Nagel edited comment on NUTCH-1483 at 11/1/12 8:55 AM:
-----------------------------------------------------------------
Thanks!
Issue with un-reversing URLs pulled out to NUTCH-1484 since it's more critical (no work-around).
Fixing the URL normalizers (and filters, see last comment) will take more time.
Btw., {{file://localhost/Documents/}} is the only legal for according to [RFC 1738|http://tools.ietf.org/html/rfc1738] (1994) while {{file:///Documents/}} is allowed by [RFC 3986|http://tools.ietf.org/html/rfc3986] (2005):
{quote}
the "file" URI scheme is defined so that no authority, an empty host, and "localhost" all mean the end-user's machine
{quote}
Maybe we could also make protocol-file more lazy.
was (Author: wastl-nagel):
Thanks!
Issue with un-reversing URLs pulled out to NUTCH-1484 since it's more critical (no work-around).
Fixing the URL normalizers (and filters, see last comment) will take more time.
Btw., {{file://localhost/Documents/}} is the only legal for according to [RFC 1738|http://tools.ietf.org/html/rfc1738] (1994) while {{file:///Documents/}} is allowed by [RFC 3986|http://tools.ietf.org/html/rfc3986]:
{quote}
the "file" URI scheme is defined so that no authority, an empty host, and "localhost" all mean the end-user's machine
{quote}
Maybe we could also make protocol-file more lazy.
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
> Attachments: NUTCH-1483.patch
>
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Sebastian Nagel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebastian Nagel updated NUTCH-1483:
-----------------------------------
Affects Version/s: 1.6
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0
> Reporter: Rogério Pereira Araújo
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488204#comment-13488204 ]
Rogério Pereira Araújo commented on NUTCH-1483:
-----------------------------------------------
No matter what I do, I get the error above (java.lang.ArrayIndexOutOfBoundsException: 1), even if I comment out the rule or disable the urlnormalizer-regex plugin, the same error came up.
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0
> Reporter: Rogério Pereira Araújo
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488191#comment-13488191 ]
Rogério Pereira Araújo commented on NUTCH-1483:
-----------------------------------------------
I tried with one slash, like file:/home/rogerio/Documents/ but I got another error while starting the crawl:
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.nutch.util.TableUtil.unreverseUrl(TableUtil.java:98)
at org.apache.nutch.crawl.GeneratorMapper.map(GeneratorMapper.java:51)
at org.apache.nutch.crawl.GeneratorMapper.map(GeneratorMapper.java:35)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0
> Reporter: Rogério Pereira Araújo
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Sebastian Nagel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488146#comment-13488146 ]
Sebastian Nagel commented on NUTCH-1483:
----------------------------------------
Confirmed.
The problem is caused by the rule
{code}
<!-- removes duplicate slashes -->
<regex>
<pattern>(?<!:)/{2,}</pattern>
<substitution>/</substitution>
</regex>
{code}
in {{regex-normalize.xml}}: {{file:///var/www/index.html}} is "normalized" to {{file://var/www/index.html}} which fails because with two slashes {{var}} is interpreted as host part of the URL (as in {{http://localhost/index.html}}) and a file {{/www/index.html}} does not exist (=404).
As work-around you can
* either remove (comment out) this rule,
* disable the plugin urlnormalizer-regex,
* or use URLs with only one slash ({{file:/home/user/docs/}}) as seeds (it surprisingly works)
Normalization of {{file://}} URLs needs to be improved but what is the canonical form?
- {{file:///var/}} or {{file:/var}}?
- what about {{file://C:/Documents/}}?
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0
> Reporter: Rogério Pereira Araújo
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rogério Pereira Araújo updated NUTCH-1483:
------------------------------------------
Comment: was deleted
(was: Patch applied and nutch recompiled successfully, now I don't get the exception anymore, neither the 404 error.
Unfortunately I'm still unable to crawl the filesystem, but this is another history :)
Thanks!)
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
> Attachments: NUTCH-1483.patch
>
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1483) Can't crawl filesystem with
protocol-file plugin
Posted by "Rogério Pereira Araújo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488386#comment-13488386 ]
Rogério Pereira Araújo commented on NUTCH-1483:
-----------------------------------------------
Patch applied and nutch recompiled successfully, now I don't get the exception anymore, neither the 404 error.
Unfortunately I'm still unable to crawl the filesystem, but this is another history :)
Thanks!
> Can't crawl filesystem with protocol-file plugin
> ------------------------------------------------
>
> Key: NUTCH-1483
> URL: https://issues.apache.org/jira/browse/NUTCH-1483
> Project: Nutch
> Issue Type: Bug
> Components: protocol
> Affects Versions: 1.6, 2.1
> Environment: OpenSUSE 12.1, OpenJDK 1.6.0, HBase 0.90.4
> Reporter: Rogério Pereira Araújo
> Attachments: NUTCH-1483.patch
>
>
> I tried to follow the same steps described in this wiki page:
> http://wiki.apache.org/nutch/IntranetDocumentSearch
> I made all required changes on regex-urlfilter.txt and added the following entry in my seed file:
> file:///home/rogerio/Documents/
> The permissions are ok, I'm running nutch with the same user as folder owner, so nutch has all the required permissions, unfortunately I'm getting the following error:
> org.apache.nutch.protocol.file.FileError: File Error: 404
> at org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:105)
> at org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:514)
> fetch of file://home/rogerio/Documents/ failed with: org.apache.nutch.protocol.file.FileError: File Error: 404
> Why the logs are showing file://home/rogerio/Documents/ instead of file:///home/rogerio/Documents/ ???
> Note: The regex-urlfilter entry only works as expected if I add the entry
> +^file://home/rogerio/Documents/ instead of +^file:///home/rogerio/Documents/ as wiki says.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira