Re : Vertex Parallelism

[Darion Yaphet <da...@gmail.com>]

Hi Madhu :

I try to create a InputFormat which could control split size by parallel .

This is for your reference . :)

public class SizeControlInputFormat extends InputFormat<Text, Text> {

    private int parallel;

    public SizeControlInputFormat(int parallel) {
        this.parallel = parallel;
    }

    @Override
    public List<InputSplit> getSplits(JobContext jobContext)
            throws IOException, InterruptedException {
        List<InputSplit> splits = new ArrayList<>(parallel);
        Path inputPath = FileInputFormat.getInputPaths(jobContext)[0];
        Configuration config = jobContext.getConfiguration();
        long fileSize =
FileSystem.get(config).getContentSummary(inputPath).getLength();
        long splitSize = fileSize / parallel;
        for (int index = 0; index < parallel; index++) {
            splits.add(new FileSplit(inputPath, index * splitSize,
splitSize,
                    (String[]) null));
        }

        return null;
    }

    @Override
    public RecordReader<Text, Text> createRecordReader(InputSplit
inputSplit,
                                                       TaskAttemptContext
taskAttemptContext)
            throws IOException, InterruptedException {

        return null;
    }
}


-- 

long is the way and hard  that out of Hell leads up to light