You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by camelia c <ca...@yahoo.com> on 2014/06/26 18:21:00 UTC
Question about org.apache.hadoop.conf.Configuration.unset
Hello,
I would like to kindly ask You for advice upon the following problem that I encountered. I write the program in Scala which makes calls to the Tez DAG API. The following call :
val intermediateDataPayload= MRHelpers.createMRIntermediateDataPayload( thePlan.tezConf, classOf[Text].getName(), classOf[IntWritable].getName(), true, null, null)
which is a Scala-equivalent for Java's :
byte[] intermediateDataPayload = MRHelpers.createMRIntermediateDataPayload(tezConf, Text.class.getName(), IntWritable.class.getName(), true, null, null);
gives the following runtime error:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.unset(Ljava/lang/String;)V
at org.apache.tez.mapreduce.hadoop.MultiStageMRConfToTezTranslator.processDirectConversion(MultiStageMRConfToTezTranslator.java:273)
at org.apache.tez.mapreduce.hadoop.MultiStageMRConfToTezTranslator.convertVertexConfToTez(MultiStageMRConfToTezTranslator.java:173)
at org.apache.tez.mapreduce.hadoop.MultiStageMRConfToTezTranslator.translateVertexConfToTez(MultiStageMRConfToTezTranslator.java:127)
at org.apache.tez.mapreduce.hadoop.MRHelpers.createMRIntermediateDataPayload(MRHelpers.java:195)
at com.twitter.summingbird.tez.Tez$.buildFlow(TezPlatform.scala:298)
...
It seems that Hive also had a problem like this , not Tez-related, that they managed through a setting in hive-site.xml:
<property> <name>fs.permissions.umask-mode</name> <value>022</value> <description> Setting a value for fs.permissions.umask-mode to work around issue in HIVE-6962. It has no impact in hadoop 1.x line on hdfs operations. </description> </property>
In my case,
hadoop version
Hadoop 2.2.0
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar
and I use the tez-0.5.0-incubating-SNAPSHOT
What approach would You recommend me?
Thank You very much,
Camelia
RE: Question about org.apache.hadoop.conf.Configuration.unset
Posted by Bikas Saha <bi...@hortonworks.com>.
Can you please post the full stack trace. I think the missing method is not
Configuration.unset() (as shown below) but some other method related to
deprecation. There have been some incompatible changes in that between 2.2
and 2.4.
Can you try with Hadoop version 2.4 instead of 2.2?
*From:* camelia c [mailto:camelie_1985@yahoo.com]
*Sent:* Thursday, June 26, 2014 9:21 AM
*To:* user@tez.incubator.apache.org
*Subject:* Question about org.apache.hadoop.conf.Configuration.unset
Hello,
I would like to kindly ask You for advice upon the following problem that I
encountered. I write the program in Scala which makes calls to the Tez DAG
API. The following call :
val intermediateDataPayload= MRHelpers.createMRIntermediateDataPayload(
thePlan.tezConf, classOf[Text].getName(), classOf[IntWritable].getName(),
true, null, null)
which is a Scala-equivalent for Java's :
byte[] intermediateDataPayload =
MRHelpers.createMRIntermediateDataPayload(tezConf, Text.class.getName(),
IntWritable.class.getName(), true, null, null);
gives the following runtime error:
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.unset(Ljava/lang/String;)V
at
org.apache.tez.mapreduce.hadoop.MultiStageMRConfToTezTranslator.processDirectConversion(MultiStageMRConfToTezTranslator.java:273)
at
org.apache.tez.mapreduce.hadoop.MultiStageMRConfToTezTranslator.convertVertexConfToTez(MultiStageMRConfToTezTranslator.java:173)
at
org.apache.tez.mapreduce.hadoop.MultiStageMRConfToTezTranslator.translateVertexConfToTez(MultiStageMRConfToTezTranslator.java:127)
at
org.apache.tez.mapreduce.hadoop.MRHelpers.createMRIntermediateDataPayload(MRHelpers.java:195)
at com.twitter.summingbird.tez.Tez$.buildFlow(TezPlatform.scala:298)
...
It seems that Hive also had a problem like this , not Tez-related, that
they managed through a setting in hive-site.xml:
<property>
<name>fs.permissions.umask-mode</name>
<value>022</value>
<description> Setting a value for fs.permissions.umask-mode to
work around issue in HIVE-6962.
It has no impact in hadoop 1.x line on hdfs operations.
</description>
</property>
In my case,
hadoop version
Hadoop 2.2.0
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
>From source with checksum 79e53ce7994d1628b240f09af91e1af4
This command was run using
/usr/local/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar
and I use the tez-0.5.0-incubating-SNAPSHOT
What approach would You recommend me?
Thank You very much,
Camelia
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.