You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2015/02/25 22:52:04 UTC

[jira] [Commented] (TEZ-2144) Compressing user payload

    [ https://issues.apache.org/jira/browse/TEZ-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337288#comment-14337288 ] 

Rohini Palaniswamy commented on TEZ-2144:
-----------------------------------------

There are 3 ways to pass split information in Tez - https://issues.apache.org/jira/browse/PIG-3564?focusedCommentId=13816848&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13816848
	
Pig sets it on the DAG submission payload to AM. Calculating splits in AM is the best. But the probem is with input formats like HCat and HBase, the delegation token is fetched with getInputSplits().

{code}
tezOp.getLoaderInfo().setInputSplitInfo(MRInputHelpers.generateInputSplitsToMem(conf, false, 0));
......
......
vertex.setLocationHint(VertexLocationHint.create(tezOp.getLoaderInfo().getInputSplitInfo().getTaskLocationHints()));
            vertex.addDataSource(ld.getOperatorKey().toString(),
                    DataSourceDescriptor.create(InputDescriptor.create(MRInput.class.getName())
                          .setUserPayload(UserPayload.create(MRRuntimeProtos.MRInputUserPayloadProto.newBuilder()
                          .setConfigurationBytes(TezUtils.createByteStringFromConf(payloadConf))
                          .setSplits(tezOp.getLoaderInfo().getInputSplitInfo().getSplitsProto()).build().toByteString().asReadOnlyByteBuffer()))
                          .setHistoryText(convertToHistoryText("", payloadConf)),
                    InputInitializerDescriptor.create(MRInputSplitDistributor.class.getName()), dag.getCredentials()));
{code}

Pig on Tez HCatLoader jobs with one of the biggest hcat tables with huge number of partitions hits the RPC limit issue. Running on HDFS data which has even triple the number of splits(100K+ splits and tasks) does not hit this issue.

HCatBaseInputFormat.java:
{code}
 //Call getSplit on the InputFormat, create an
      //HCatSplit for each underlying split
      //NumSplits is 0 for our purposes
      org.apache.hadoop.mapred.InputSplit[] baseSplits = 
        inputFormat.getSplits(jobConf, 0);

      for(org.apache.hadoop.mapred.InputSplit split : baseSplits) {
        splits.add(new HCatSplit(
            partitionInfo,
            split,allCols));
      }
{code}

Each hcatSplit duplicates a lot of information - partition schema and table schema

On the Pig side, trying to workaround the problem by using the alternative of writing the split files to hdfs for big splits like this. Increasing the RPC size limit will not help as it is going to 300MB just for a day's worth of data. Weekly and monthly data scanning will make it run into GBs.

Had a discussion with [~sseth] to see if it make sense for tez to compress the input split information. It would help a lot in cases like this. Even for HDFS paths, with the same base path being repeated compression would help. He asked to create this jira to see if the compression would need to be in Tez or Pig, check if there will be any gain if each proto is compressed separately or should the entire payload be compressed




> Compressing user payload
> ------------------------
>
>                 Key: TEZ-2144
>                 URL: https://issues.apache.org/jira/browse/TEZ-2144
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Rohini Palaniswamy
>
> Pig sets the input split information in user payload and when running against a table with 10s of 1000s of partitions, DAG submission fails with 
> java.io.IOException: Requested data length 305844060 is longer than maximum
> configured RPC length 67108864



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)