Re:Re: Vertex Parallelism

["darion.yaphet" <fl...@163.com>]

Hi Madhu :<br/><br/>I try to create a InputFormat which could control split size by parallel . This is for your reference . :)<br/><br/>public class SizeControlInputFormat extends InputFormat&lt;Text, Text&gt; {<br/><br/>    private int parallel;<br/><br/>    public SizeControlInputFormat(int parallel) {<br/>        this.parallel = parallel;<br/>    }<br/><br/>    @Override<br/>    public List&lt;InputSplit&gt; getSplits(JobContext jobContext)<br/>            throws IOException, InterruptedException {<br/>        List&lt;InputSplit&gt; splits = new ArrayList&lt;&gt;(parallel);<br/>        Path inputPath = FileInputFormat.getInputPaths(jobContext)[0];<br/>        Configuration config = jobContext.getConfiguration();<br/>        long fileSize = FileSystem.get(config).getContentSummary(inputPath).getLength();<br/>        long splitSize = fileSize / parallel;<br/>        for (int index = 0; index &lt; parallel; index++) {<br/>            splits.add(new FileSplit(inputPath, index * splitSize, splitSize,<br/>                    (String[]) null));<br/>        }<br/><br/>        return null;<br/>    }<br/><br/>    @Override<br/>    public RecordReader&lt;Text, Text&gt; createRecordReader(InputSplit inputSplit,<br/>                                                       TaskAttemptContext taskAttemptContext)<br/>            throws IOException, InterruptedException {<br/><br/>        return null;<br/>    }<br/>}
在 2016-11-01 07:05:56，"Hitesh Shah" <hi...@apache.org> 写道：
>I suggest writing a custom InputFormat or modifying your existing InputFormat to generate more splits and at the same time, disable splits grouping for the vertex in question to ensure that you get the high level of parallelism that you want to achieve.
>
>The log snippet is just indicating that vertex had been setup with -1 tasks as the splits are being calculated in the AM and that the vertex parallelism will be set via the initializer/controller (based on the splits from the Input Format).
>
>— Hitesh
>
>> On Oct 31, 2016, at 3:33 PM, Madhusudan Ramanna <m....@ymail.com> wrote:
>> 
>> Hello Tez team,
>> 
>> We have a native Tez application.  The first vertex in the graph is a downloader.  This vertex takes a CSV or sequence file that contains the "urls" as input, downloads content and passes the content on to the next vertex.  This input to vertex is smaller than the min split size.   However, we'd like to have more than one task for running for this vertex to help throughput. How do we set the tasks on this particular vertex to be greater than one ?  Of course for other vertices in the graph,  number of tasks as computed by data size fits perfectly fine. 
>> 
>> Currently, we're seeing this in the logs:
>> 
>> >>>>>
>> 
>> Root Inputs exist for Vertex: download : {_initial={InputName=_initial}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
>> Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1477944280627_0004_1_00 [download]
>> Vertex will initialize from input initializer. vertex_1477944280627_0004_1_00 [download]
>> <<<<<
>> 
>> 
>> 
>> Thanks for your help !
>> 
>> Madhu
>> 
>> 
>> 
>