You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Rong-En Fan (JIRA)" <ji...@apache.org> on 2008/04/02 06:41:25 UTC
[jira] Created: (HADOOP-3152) Make index interval configuable when
using MapFileOutputFormat for map-reduce job
Make index interval configuable when using MapFileOutputFormat for map-reduce job
---------------------------------------------------------------------------------
Key: HADOOP-3152
URL: https://issues.apache.org/jira/browse/HADOOP-3152
Project: Hadoop Core
Issue Type: Bug
Components: io
Affects Versions: 0.16.0
Environment: ubuntu, 64bit, jdk1.6
Reporter: Rong-En Fan
Attachments: mapfile.diff
Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
thread title "MapFile and MapFileOutputFormat". Currently, there is no way
to change the index interval for the output MapFile in a map-reduce job.
As suggested, adding a static method MapFile(Configuration, int) to set the
index interval and stores in Configuration, then MapFile.Writer constructor
reads the setting from configuration may be a good idea.
I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3152) Make index interval configuable when
using MapFileOutputFormat for map-reduce job
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rong-En Fan updated HADOOP-3152:
--------------------------------
Attachment: mapfile.diff
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Attachments: mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3152) Make index interval configuable
when using MapFileOutputFormat for map-reduce job
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584939#action_12584939 ]
Hadoop QA commented on HADOOP-3152:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379100/mapfile.diff
against trunk revision 643282.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2129/console
This message is automatically generated.
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Fix For: 0.17.0
>
> Attachments: mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3152) Make index interval configuable
when using MapFileOutputFormat for map-reduce job
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585426#action_12585426 ]
Rong-En Fan commented on HADOOP-3152:
-------------------------------------
New patch is attached with unit test. Please kindly review it. Thanks!
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Fix For: 0.17.0
>
> Attachments: mapfile.diff, mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3152) Make index interval configuable
when using MapFileOutputFormat for map-reduce job
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584943#action_12584943 ]
Rong-En Fan commented on HADOOP-3152:
-------------------------------------
There is not existing test for index interval in TestMapFile.java unit test.
So I did not add a new one for this. If this is necessary, then I will try to write
one for it.
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Fix For: 0.17.0
>
> Attachments: mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3152) Make index interval configuable when
using MapFileOutputFormat for map-reduce job
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-3152:
---------------------------------
Status: Patch Available (was: Open)
Looks good to me!
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Fix For: 0.17.0
>
> Attachments: mapfile.diff, mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3152) Make index interval configuable when
using MapFileOutputFormat for map-reduce job
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-3152:
---------------------------------
Resolution: Fixed
Release Note: Add a static method MapFile#setIndexInterval(Configuration, int interval) so that MapReduce jobs that use MapFileOutputFormat can set the index interval.
Status: Resolved (was: Patch Available)
I committed this.
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Fix For: 0.17.0
>
> Attachments: mapfile.diff, mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3152) Make index interval configuable
when using MapFileOutputFormat for map-reduce job
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12585983#action_12585983 ]
Hudson commented on HADOOP-3152:
--------------------------------
Integrated in Hadoop-trunk #451 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/451/])
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 0.17.0
>
> Attachments: mapfile.diff, mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3152) Make index interval configuable when
using MapFileOutputFormat for map-reduce job
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-3152:
---------------------------------
Fix Version/s: 0.17.0
Assignee: Doug Cutting
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
+1 This looks good to me.
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Fix For: 0.17.0
>
> Attachments: mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3152) Make index interval configuable when
using MapFileOutputFormat for map-reduce job
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3152:
----------------------------------
Status: Open (was: Patch Available)
Please do write one. Thanks!
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Fix For: 0.17.0
>
> Attachments: mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3152) Make index interval configuable when
using MapFileOutputFormat for map-reduce job
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-3152:
---------------------------------
Priority: Minor (was: Major)
Issue Type: New Feature (was: Bug)
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: New Feature
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Priority: Minor
> Fix For: 0.17.0
>
> Attachments: mapfile.diff, mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3152) Make index interval configuable
when using MapFileOutputFormat for map-reduce job
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584403#action_12584403 ]
Rong-En Fan commented on HADOOP-3152:
-------------------------------------
Please kindly do code review for this patch. Thanks!
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Attachments: mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3152) Make index interval configuable when
using MapFileOutputFormat for map-reduce job
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rong-En Fan updated HADOOP-3152:
--------------------------------
Attachment: mapfile.diff
same patch, but includes unit test
> Make index interval configuable when using MapFileOutputFormat for map-reduce job
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3152
> URL: https://issues.apache.org/jira/browse/HADOOP-3152
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.0
> Environment: ubuntu, 64bit, jdk1.6
> Reporter: Rong-En Fan
> Assignee: Doug Cutting
> Fix For: 0.17.0
>
> Attachments: mapfile.diff, mapfile.diff
>
>
> Per discussion with Doug Cutting on hadoop user mailing around Mar 21,
> thread title "MapFile and MapFileOutputFormat". Currently, there is no way
> to change the index interval for the output MapFile in a map-reduce job.
> As suggested, adding a static method MapFile(Configuration, int) to set the
> index interval and stores in Configuration, then MapFile.Writer constructor
> reads the setting from configuration may be a good idea.
> I also noticed that Hbase did similar things in HBASE-364.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.