You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "wang qiang (Jira)" <ji...@apache.org> on 2020/03/10 02:02:00 UTC
[jira] [Updated] (TEZ-4133) key class implments writableComparable
and configurable use default configuration
[ https://issues.apache.org/jira/browse/TEZ-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
wang qiang updated TEZ-4133:
----------------------------
Description:
key class implments writableComparable and configurable ,setConf be invoked with a default configuration,it should like mapreduce(org.apache.hadoop.mapred.JobConf#getOutputKeyComparator)
{code:java}
//org.apache.tez.runtime.library.common.ConfigUtils#getIntermediateOutputKeyComparator
public static <K> RawComparator<K> getIntermediateOutputKeyComparator(Configuration conf) {
Class<? extends RawComparator> theClass = conf.getClass(
TezRuntimeConfiguration.TEZ_RUNTIME_KEY_COMPARATOR_CLASS, null,
RawComparator.class);
if (theClass != null)
return ReflectionUtils.newInstance(theClass, conf);
// aybe change to
// return WritableComparator.get(getIntermediateOutputKeyClass(conf).asSubclass( //WritableComparable.class),conf);
return WritableComparator.get(getIntermediateOutputKeyClass(conf).asSubclass(
WritableComparable.class));
}
public static <K> RawComparator<K> getIntermediateInputKeyComparator(Configuration conf) {
Class<? extends RawComparator> theClass = conf.getClass(
TezRuntimeConfiguration.TEZ_RUNTIME_KEY_COMPARATOR_CLASS, null,
RawComparator.class);
if (theClass != null)
return ReflectionUtils.newInstance(theClass, conf);
// maybe change to
// return WritableComparator.get(getIntermediateInputKeyClass(conf).asSubclass( //WritableComparable.class),conf);
return WritableComparator.get(getIntermediateInputKeyClass(conf).asSubclass(
WritableComparable.class));
}
{code}
was:
key class implments writableComparable and configurable ,setConf be invoked with a default configuration,it should like mapreduce(org.apache.hadoop.mapred.JobConf#getOutputKeyComparator)
{code:java}
//代码占位符
public static <K> RawComparator<K> getIntermediateOutputKeyComparator(Configuration conf) {
Class<? extends RawComparator> theClass = conf.getClass(
TezRuntimeConfiguration.TEZ_RUNTIME_KEY_COMPARATOR_CLASS, null,
RawComparator.class);
if (theClass != null)
return ReflectionUtils.newInstance(theClass, conf);
// aybe change to
// return WritableComparator.get(getIntermediateOutputKeyClass(conf).asSubclass( //WritableComparable.class),conf);
return WritableComparator.get(getIntermediateOutputKeyClass(conf).asSubclass(
WritableComparable.class));
}
public static <K> RawComparator<K> getIntermediateInputKeyComparator(Configuration conf) {
Class<? extends RawComparator> theClass = conf.getClass(
TezRuntimeConfiguration.TEZ_RUNTIME_KEY_COMPARATOR_CLASS, null,
RawComparator.class);
if (theClass != null)
return ReflectionUtils.newInstance(theClass, conf);
// maybe change to
// return WritableComparator.get(getIntermediateInputKeyClass(conf).asSubclass( //WritableComparable.class),conf);
return WritableComparator.get(getIntermediateInputKeyClass(conf).asSubclass(
WritableComparable.class));
}
{code}
> key class implments writableComparable and configurable use default configuration
> ---------------------------------------------------------------------------------
>
> Key: TEZ-4133
> URL: https://issues.apache.org/jira/browse/TEZ-4133
> Project: Apache Tez
> Issue Type: Bug
> Reporter: wang qiang
> Priority: Major
>
> key class implments writableComparable and configurable ,setConf be invoked with a default configuration,it should like mapreduce(org.apache.hadoop.mapred.JobConf#getOutputKeyComparator)
> {code:java}
> //org.apache.tez.runtime.library.common.ConfigUtils#getIntermediateOutputKeyComparator
> public static <K> RawComparator<K> getIntermediateOutputKeyComparator(Configuration conf) {
> Class<? extends RawComparator> theClass = conf.getClass(
> TezRuntimeConfiguration.TEZ_RUNTIME_KEY_COMPARATOR_CLASS, null,
> RawComparator.class);
> if (theClass != null)
> return ReflectionUtils.newInstance(theClass, conf);
> // aybe change to
> // return WritableComparator.get(getIntermediateOutputKeyClass(conf).asSubclass( //WritableComparable.class),conf);
> return WritableComparator.get(getIntermediateOutputKeyClass(conf).asSubclass(
> WritableComparable.class));
> }
> public static <K> RawComparator<K> getIntermediateInputKeyComparator(Configuration conf) {
> Class<? extends RawComparator> theClass = conf.getClass(
> TezRuntimeConfiguration.TEZ_RUNTIME_KEY_COMPARATOR_CLASS, null,
> RawComparator.class);
> if (theClass != null)
> return ReflectionUtils.newInstance(theClass, conf);
> // maybe change to
> // return WritableComparator.get(getIntermediateInputKeyClass(conf).asSubclass( //WritableComparable.class),conf);
> return WritableComparator.get(getIntermediateInputKeyClass(conf).asSubclass(
> WritableComparable.class));
> }
> {code}
>
>
>
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)