[jira] [Commented] (HAMA-781) Setting partition split fails in local mode when file size is big and has a runtime partition (HashParitioner)

["Ikhtiyor Ahmedov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HAMA-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719247#comment-13719247 ] 

Ikhtiyor Ahmedov commented on HAMA-781:
---------------------------------------

Same code affects when multiple inputs given as input.
Code: 
{quote}
        // set partitionID to rawSplit
        if (split.getClass().getName().equals(FileSplit.class.getName())
            && job.getConfiguration().get(Constants.RUNTIME_PARTITIONING_CLASS) != null
            && job.get("bsp.partitioning.runner.job") == null) {
          LOG.debug(((FileSplit) split).getPath().getName());
          String[] extractPartitionID = ((FileSplit) split).getPath().getName()
              .split("[-]");
          rawSplit.setPartitionID(Integer.parseInt(extractPartitionID[1]));
        }
{quote}
Exception:
{quote}java.lang.ArrayIndexOutOfBoundsException: 1
	at org.apache.hama.bsp.BSPJobClient.writeSplits(BSPJobClient.java:566)
	at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:342)
	at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:293)
	at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:229)
	at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:236)
	at org.apache.hama.examples.OnlineCF.main(OnlineCF.java:427)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
	at org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:44)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hama.util.RunJar.main(RunJar.java:146)
{quote}
Example fail cases:
1) When input is non-hdfs format (part-0000, part-0001) and size is big (usually from local filesystem)
2) When input is given as multiple files in local mode: 
{quote}SequenceFileInputFormat.addInputPaths(job, "/tmp/test.seq,/tmp/test2.seq,/tmp/test3.seq");{quote}
                
> Setting partition split fails in local mode when file size is big and has a runtime partition (HashParitioner)
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HAMA-781
>                 URL: https://issues.apache.org/jira/browse/HAMA-781
>             Project: Hama
>          Issue Type: Bug
>          Components: bsp core
>            Reporter: Ikhtiyor Ahmedov
>            Priority: Minor
>         Attachments: HAMA-781.patch
>
>
> when input partitioner set to HashPartitioner and file size is big in local mode; in line 566 of BSPJobClient.java throws index out of bound exception.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira