[jira] [Commented] (TEZ-1080) Configuration for non MR based Inputs/Outputs

["Siddharth Seth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033143#comment-14033143 ] 

Siddharth Seth commented on TEZ-1080:
-------------------------------------

Usage examples

Complex, configuring multiple properties.
{code}
 SortedPartitionedPairConf conf = SortedPartitionedPairConf.newBuilder("keyClass", "valClass")
        .setKeyComparatorClass("comparatorClass")
        .enableCompression(null)
        .configureOutput("partitionerClassName")
            .setSortBufferSize(1024)
            .setSorterNumThreads(1)
            .build()
        .configureInput()
            .setCombiner("Combiner", Map<String, String> combinerProperties)
            .setShuffleBufferFraction(0.25f)
            .build()
        .setAdditionalConf("key", "val")
        .setAdditionalCons(Map<String, String>)
        .build();

  conf.getInputPayload();
  conf.getOutputPayload();
{code}

Simple, relying primarily on defaults
{code}
 SortedPartitionedPairConf conf = SortedPartitionedPairConf.newBuilder("keyClass", "valClass")
     .configureOutput("partitionerClassName").build()
      .configureInput().build()
  .build();

  conf.getInputPayload();
  conf.getOutputPayload();
{code}

Optionally, these configuration objects could generated the actual Edge to be used in the DAGPlan.
  Edge conf.createEdge(... ...) 

Combiners, Partitioners etc would receive separate key-value pairs, if they need additional configuration.

The current method of relying on a configuration would no longer work.

[~rohini], [~hagleitn] - any feedback on this API ?

> Configuration for non MR based Inputs/Outputs
> ---------------------------------------------
>
>                 Key: TEZ-1080
>                 URL: https://issues.apache.org/jira/browse/TEZ-1080
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>         Attachments: TEZ-1080.wip.1.txt, TEZ-1080.wip.2.txt
>
>
> De-link configuration from MRHelpers (except for the YARNRunner case), and allow for these to be configured easily - exposing necessary setters / getters without having to rely on config keys.



--
This message was sent by Atlassian JIRA
(v6.2#6252)