You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Dogacan Güney (JIRA)" <ji...@apache.org> on 2006/12/15 14:27:20 UTC
[jira] Created: (NUTCH-417) After upgrade to hadoop-0.9.1, parsing
and indexing doesn't work.
After upgrade to hadoop-0.9.1, parsing and indexing doesn't work.
-----------------------------------------------------------------
Key: NUTCH-417
URL: http://issues.apache.org/jira/browse/NUTCH-417
Project: Nutch
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Dogacan Güney
If you parse while fetching then it is fine, but if you run parse as a different job, it creates an essentially empty parse_data directory(which has index files, but doesn't have data files). I am not sure why this is happening.
Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The parameter fs seems to be an instance of PhasedFileSystem which throws exceptions on delete and {start,complete}LocalOutput.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-417) After upgrade to hadoop-0.9.1,
parsing and indexing doesn't work.
Posted by "Dogacan Güney (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-417?page=comments#action_12458794 ]
Dogacan Güney commented on NUTCH-417:
-------------------------------------
Patch for indexer. Instead of using the FileSystem coming from getRecordWriter, use FileSystem.get(job) to get the file system. (btw: probably not the best way of doing this)
> After upgrade to hadoop-0.9.1, parsing and indexing doesn't work.
> -----------------------------------------------------------------
>
> Key: NUTCH-417
> URL: http://issues.apache.org/jira/browse/NUTCH-417
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Dogacan Güney
> Attachments: index.patch
>
>
> If you parse while fetching then it is fine, but if you run parse as a different job, it creates an essentially empty parse_data directory(which has index files, but doesn't have data files). I am not sure why this is happening.
> Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The parameter fs seems to be an instance of PhasedFileSystem which throws exceptions on delete and {start,complete}LocalOutput.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-417) After upgrade to hadoop-0.9.1, parsing
and indexing doesn't work.
Posted by "Dogacan Güney (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-417?page=all ]
Dogacan Güney updated NUTCH-417:
--------------------------------
Attachment: index.patch
> After upgrade to hadoop-0.9.1, parsing and indexing doesn't work.
> -----------------------------------------------------------------
>
> Key: NUTCH-417
> URL: http://issues.apache.org/jira/browse/NUTCH-417
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Dogacan Güney
> Attachments: index.patch
>
>
> If you parse while fetching then it is fine, but if you run parse as a different job, it creates an essentially empty parse_data directory(which has index files, but doesn't have data files). I am not sure why this is happening.
> Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The parameter fs seems to be an instance of PhasedFileSystem which throws exceptions on delete and {start,complete}LocalOutput.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Closed: (NUTCH-417) After upgrade to hadoop-0.9.1, parsing
and indexing doesn't work.
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki closed NUTCH-417.
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.9.0
Assignee: Andrzej Bialecki
Fixed as a part of upgrade to Hadoop 0.12.2
> After upgrade to hadoop-0.9.1, parsing and indexing doesn't work.
> -----------------------------------------------------------------
>
> Key: NUTCH-417
> URL: https://issues.apache.org/jira/browse/NUTCH-417
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Doğacan Güney
> Assigned To: Andrzej Bialecki
> Fix For: 0.9.0
>
> Attachments: index.patch
>
>
> If you parse while fetching then it is fine, but if you run parse as a different job, it creates an essentially empty parse_data directory(which has index files, but doesn't have data files). I am not sure why this is happening.
> Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The parameter fs seems to be an instance of PhasedFileSystem which throws exceptions on delete and {start,complete}LocalOutput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-417) After upgrade to hadoop-0.9.1,
parsing and indexing doesn't work.
Posted by "Dogacan Güney (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-417?page=comments#action_12458811 ]
Dogacan Güney commented on NUTCH-417:
-------------------------------------
Setting speculative execution to false also fixes my problem with parser. Thank you for the quick answer. I guess you can close this issue now.
> After upgrade to hadoop-0.9.1, parsing and indexing doesn't work.
> -----------------------------------------------------------------
>
> Key: NUTCH-417
> URL: http://issues.apache.org/jira/browse/NUTCH-417
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Dogacan Güney
> Attachments: index.patch
>
>
> If you parse while fetching then it is fine, but if you run parse as a different job, it creates an essentially empty parse_data directory(which has index files, but doesn't have data files). I am not sure why this is happening.
> Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The parameter fs seems to be an instance of PhasedFileSystem which throws exceptions on delete and {start,complete}LocalOutput.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-417) After upgrade to hadoop-0.9.1,
parsing and indexing doesn't work.
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-417?page=comments#action_12458800 ]
Andrzej Bialecki commented on NUTCH-417:
-----------------------------------------
Regarding the issue in Indexer: a word of warning - you are running with mapred speculative execution set to true. Speculative execution is still not recommended with hadoop 0.9.1, so please set it to false for now.
> After upgrade to hadoop-0.9.1, parsing and indexing doesn't work.
> -----------------------------------------------------------------
>
> Key: NUTCH-417
> URL: http://issues.apache.org/jira/browse/NUTCH-417
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Dogacan Güney
> Attachments: index.patch
>
>
> If you parse while fetching then it is fine, but if you run parse as a different job, it creates an essentially empty parse_data directory(which has index files, but doesn't have data files). I am not sure why this is happening.
> Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The parameter fs seems to be an instance of PhasedFileSystem which throws exceptions on delete and {start,complete}LocalOutput.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-417) After upgrade to hadoop-0.9.1,
parsing and indexing doesn't work.
Posted by "Sean Dean (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-417?page=comments#action_12459073 ]
Sean Dean commented on NUTCH-417:
---------------------------------
Speculative execution is now off by default with Hadoop 0.9.2 as per issue HADOOP-827. Since there was only two other fixes with that distribution, neither of which should effect Nutch in a bad way can that be updated in trunk?
> After upgrade to hadoop-0.9.1, parsing and indexing doesn't work.
> -----------------------------------------------------------------
>
> Key: NUTCH-417
> URL: http://issues.apache.org/jira/browse/NUTCH-417
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Dogacan Güney
> Attachments: index.patch
>
>
> If you parse while fetching then it is fine, but if you run parse as a different job, it creates an essentially empty parse_data directory(which has index files, but doesn't have data files). I am not sure why this is happening.
> Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The parameter fs seems to be an instance of PhasedFileSystem which throws exceptions on delete and {start,complete}LocalOutput.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-417) After upgrade to hadoop-0.9.1,
parsing and indexing doesn't work.
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-417?page=comments#action_12459079 ]
Andrzej Bialecki commented on NUTCH-417:
-----------------------------------------
Yes, I'm going to do exactly this in a few days - I thought about putting this property in hadoop-site.xml in Nutch, but this way it's even better.
> After upgrade to hadoop-0.9.1, parsing and indexing doesn't work.
> -----------------------------------------------------------------
>
> Key: NUTCH-417
> URL: http://issues.apache.org/jira/browse/NUTCH-417
> Project: Nutch
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Dogacan Güney
> Attachments: index.patch
>
>
> If you parse while fetching then it is fine, but if you run parse as a different job, it creates an essentially empty parse_data directory(which has index files, but doesn't have data files). I am not sure why this is happening.
> Also, indexing fails at Indexer.OutputFormat.getRecordWriter. The parameter fs seems to be an instance of PhasedFileSystem which throws exceptions on delete and {start,complete}LocalOutput.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira