You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2011/07/08 07:29:16 UTC
[jira] [Created] (MAPREDUCE-2660) MR-279:
org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws
ArrayStoreException
MR-279: org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws ArrayStoreException
----------------------------------------------------------------------------------------------
Key: MAPREDUCE-2660
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2660
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Amareshwari Sriramadasu
In branch MR-279, org.apache.hadoop.mapred.getSplits() throws RuntimeException:ArrayStoreException
The following code in trunk:
{noformat}
public InputSplit[] getSplits(JobConf job, int numSplits)
throws IOException {
List<org.apache.hadoop.mapreduce.InputSplit> newStyleSplits =
super.getSplits(new Job(job));
InputSplit[] ret = new InputSplit[newStyleSplits.size()];
for(int pos = 0; pos < newStyleSplits.size(); ++pos) {
org.apache.hadoop.mapreduce.lib.input.CombineFileSplit newStyleSplit =
(org.apache.hadoop.mapreduce.lib.input.CombineFileSplit) newStyleSplits.get(pos);
ret[pos] = new CombineFileSplit(job, newStyleSplit.getPaths(),
newStyleSplit.getStartOffsets(), newStyleSplit.getLengths(),
newStyleSplit.getLocations());
}
return ret;
}
{noformat}
got changed to
{noformat}
public InputSplit[] getSplits(JobConf job, int numSplits)
throws IOException {
return super.getSplits(new Job(job)).toArray(new InputSplit[0]);
}
{noformat}
Code in trunk works fine. We should change the code in MR-279 to the same in trunk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2660) MR-279:
org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws
ArrayStoreException
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064353#comment-13064353 ]
Amareshwari Sriramadasu commented on MAPREDUCE-2660:
----------------------------------------------------
Thanks Vinod. The patch works for me.
> MR-279: org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws ArrayStoreException
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2660
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2660
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: MAPREDUCE-2571.patch, MAPREDUCE-2660-MR-279.patch
>
>
> In branch MR-279, org.apache.hadoop.mapred.getSplits() throws RuntimeException:ArrayStoreException
> The following code in trunk:
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> List<org.apache.hadoop.mapreduce.InputSplit> newStyleSplits =
> super.getSplits(new Job(job));
> InputSplit[] ret = new InputSplit[newStyleSplits.size()];
> for(int pos = 0; pos < newStyleSplits.size(); ++pos) {
> org.apache.hadoop.mapreduce.lib.input.CombineFileSplit newStyleSplit =
> (org.apache.hadoop.mapreduce.lib.input.CombineFileSplit) newStyleSplits.get(pos);
> ret[pos] = new CombineFileSplit(job, newStyleSplit.getPaths(),
> newStyleSplit.getStartOffsets(), newStyleSplit.getLengths(),
> newStyleSplit.getLocations());
> }
> return ret;
> }
> {noformat}
> got changed to
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> return super.getSplits(new Job(job)).toArray(new InputSplit[0]);
> }
> {noformat}
> Code in trunk works fine. We should change the code in MR-279 to the same in trunk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2660) MR-279:
org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws
ArrayStoreException
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2660:
-----------------------------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
The issue got fixed in 0.22 and trunk by MAPREDUCE-2571. We can push it to MR-279 branch if required.
> MR-279: org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws ArrayStoreException
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2660
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2660
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: MAPREDUCE-2660-MR-279.patch
>
>
> In branch MR-279, org.apache.hadoop.mapred.getSplits() throws RuntimeException:ArrayStoreException
> The following code in trunk:
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> List<org.apache.hadoop.mapreduce.InputSplit> newStyleSplits =
> super.getSplits(new Job(job));
> InputSplit[] ret = new InputSplit[newStyleSplits.size()];
> for(int pos = 0; pos < newStyleSplits.size(); ++pos) {
> org.apache.hadoop.mapreduce.lib.input.CombineFileSplit newStyleSplit =
> (org.apache.hadoop.mapreduce.lib.input.CombineFileSplit) newStyleSplits.get(pos);
> ret[pos] = new CombineFileSplit(job, newStyleSplit.getPaths(),
> newStyleSplit.getStartOffsets(), newStyleSplit.getLengths(),
> newStyleSplit.getLocations());
> }
> return ret;
> }
> {noformat}
> got changed to
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> return super.getSplits(new Job(job)).toArray(new InputSplit[0]);
> }
> {noformat}
> Code in trunk works fine. We should change the code in MR-279 to the same in trunk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2660) MR-279:
org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws
ArrayStoreException
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2660:
-----------------------------------------------
Attachment: MAPREDUCE-2571.patch
Uploading a patch for MR-279 - this is same as the patch that got committed at MAPREDUCE-2571, but with different directory structure.
Amareshwari, can you please do a sanity check and see if it works for you?
> MR-279: org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws ArrayStoreException
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2660
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2660
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: MAPREDUCE-2571.patch, MAPREDUCE-2660-MR-279.patch
>
>
> In branch MR-279, org.apache.hadoop.mapred.getSplits() throws RuntimeException:ArrayStoreException
> The following code in trunk:
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> List<org.apache.hadoop.mapreduce.InputSplit> newStyleSplits =
> super.getSplits(new Job(job));
> InputSplit[] ret = new InputSplit[newStyleSplits.size()];
> for(int pos = 0; pos < newStyleSplits.size(); ++pos) {
> org.apache.hadoop.mapreduce.lib.input.CombineFileSplit newStyleSplit =
> (org.apache.hadoop.mapreduce.lib.input.CombineFileSplit) newStyleSplits.get(pos);
> ret[pos] = new CombineFileSplit(job, newStyleSplit.getPaths(),
> newStyleSplit.getStartOffsets(), newStyleSplit.getLengths(),
> newStyleSplit.getLocations());
> }
> return ret;
> }
> {noformat}
> got changed to
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> return super.getSplits(new Job(job)).toArray(new InputSplit[0]);
> }
> {noformat}
> Code in trunk works fine. We should change the code in MR-279 to the same in trunk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2660) MR-279:
org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws
ArrayStoreException
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2660:
-----------------------------------------------
Attachment: MAPREDUCE-2660-MR-279.patch
Patch with the changes.
> MR-279: org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws ArrayStoreException
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2660
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2660
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Amareshwari Sriramadasu
> Attachments: MAPREDUCE-2660-MR-279.patch
>
>
> In branch MR-279, org.apache.hadoop.mapred.getSplits() throws RuntimeException:ArrayStoreException
> The following code in trunk:
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> List<org.apache.hadoop.mapreduce.InputSplit> newStyleSplits =
> super.getSplits(new Job(job));
> InputSplit[] ret = new InputSplit[newStyleSplits.size()];
> for(int pos = 0; pos < newStyleSplits.size(); ++pos) {
> org.apache.hadoop.mapreduce.lib.input.CombineFileSplit newStyleSplit =
> (org.apache.hadoop.mapreduce.lib.input.CombineFileSplit) newStyleSplits.get(pos);
> ret[pos] = new CombineFileSplit(job, newStyleSplit.getPaths(),
> newStyleSplit.getStartOffsets(), newStyleSplit.getLengths(),
> newStyleSplit.getLocations());
> }
> return ret;
> }
> {noformat}
> got changed to
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> return super.getSplits(new Job(job)).toArray(new InputSplit[0]);
> }
> {noformat}
> Code in trunk works fine. We should change the code in MR-279 to the same in trunk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2660) MR-279:
org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws
ArrayStoreException
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-2660:
-----------------------------------------------
Assignee: Amareshwari Sriramadasu
Status: Patch Available (was: Open)
> MR-279: org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws ArrayStoreException
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2660
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2660
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: MAPREDUCE-2660-MR-279.patch
>
>
> In branch MR-279, org.apache.hadoop.mapred.getSplits() throws RuntimeException:ArrayStoreException
> The following code in trunk:
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> List<org.apache.hadoop.mapreduce.InputSplit> newStyleSplits =
> super.getSplits(new Job(job));
> InputSplit[] ret = new InputSplit[newStyleSplits.size()];
> for(int pos = 0; pos < newStyleSplits.size(); ++pos) {
> org.apache.hadoop.mapreduce.lib.input.CombineFileSplit newStyleSplit =
> (org.apache.hadoop.mapreduce.lib.input.CombineFileSplit) newStyleSplits.get(pos);
> ret[pos] = new CombineFileSplit(job, newStyleSplit.getPaths(),
> newStyleSplit.getStartOffsets(), newStyleSplit.getLengths(),
> newStyleSplit.getLocations());
> }
> return ret;
> }
> {noformat}
> got changed to
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> return super.getSplits(new Job(job)).toArray(new InputSplit[0]);
> }
> {noformat}
> Code in trunk works fine. We should change the code in MR-279 to the same in trunk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2660) MR-279:
org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws
ArrayStoreException
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061822#comment-13061822 ]
Hadoop QA commented on MAPREDUCE-2660:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12485703/MAPREDUCE-2660-MR-279.patch
against trunk revision 1144097.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/444//console
This message is automatically generated.
> MR-279: org.apache.hadoop.mapred.CombineFileInputFormat.getSplits() throws ArrayStoreException
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2660
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2660
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: MAPREDUCE-2660-MR-279.patch
>
>
> In branch MR-279, org.apache.hadoop.mapred.getSplits() throws RuntimeException:ArrayStoreException
> The following code in trunk:
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> List<org.apache.hadoop.mapreduce.InputSplit> newStyleSplits =
> super.getSplits(new Job(job));
> InputSplit[] ret = new InputSplit[newStyleSplits.size()];
> for(int pos = 0; pos < newStyleSplits.size(); ++pos) {
> org.apache.hadoop.mapreduce.lib.input.CombineFileSplit newStyleSplit =
> (org.apache.hadoop.mapreduce.lib.input.CombineFileSplit) newStyleSplits.get(pos);
> ret[pos] = new CombineFileSplit(job, newStyleSplit.getPaths(),
> newStyleSplit.getStartOffsets(), newStyleSplit.getLengths(),
> newStyleSplit.getLocations());
> }
> return ret;
> }
> {noformat}
> got changed to
> {noformat}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> return super.getSplits(new Job(job)).toArray(new InputSplit[0]);
> }
> {noformat}
> Code in trunk works fine. We should change the code in MR-279 to the same in trunk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira