You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Rajat Jain <ra...@gmail.com> on 2016/02/03 07:19:05 UTC
Running using Tez minimal package
In our current hive-on-tez deployment, we use the tez regular package which
has hadoop-* jars in tez/lib.
When we switched to a tez-minimal and started getting errors while
launching containers, presumably because hadoop-* jars are not present in
the container launch classpath. How can this be fixed?
Thanks,
Rajat
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/service/AbstractService
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.service.AbstractService
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more
Re: Running using Tez minimal package
Posted by Rajat Jain <ra...@gmail.com>.
Thanks, Sid. Setting this parameter solves this problem. We usually do not
care about rolling deploys on our end. In our cloud based model, we just do
a rolling restart of the cluster
<https://www.qubole.com/blog/product/drag-n-drop-upgrades-of-hadoop-spark-and-presto-clusters/>
with the updated libraries -- it has given us a far smoother experience.
Since our hadoop and tez libraries are tightly linked in an Amazon AMI, we
like to keep the tez library lean.
On 3 February 2016 at 12:27, Siddharth Seth <ss...@apache.org> wrote:
> Setting tez.use.cluster.hadoop-libs to true is one option. This will try
> picking up the classpath from the cluster itself - I believe a value
> configured in yarn-site.xml. The drawback to this though is a that job
> which are running during a rolling upgrade of the cluster can run into
> issues - 1) Different tasks running with different versions of the
> hadoop-client, 2) jars being replaced from under a running process.
> Alternately, if you're trying to use a custom version of hadoop-libraries,
> these can be place in HDFS and setup via 'tez.aux.uris'
>
> The suggested deployment is with the regular package though. Is there a
> specific reason for not making use of this. If a custom version of hadoop
> is required, the package can be built by providing the -Dhadoop.version
> flag to maven.
>
> - Sid
>
>
>
>
>
> On Tue, Feb 2, 2016 at 10:19 PM, Rajat Jain <ra...@gmail.com> wrote:
>
>> In our current hive-on-tez deployment, we use the tez regular package
>> which has hadoop-* jars in tez/lib.
>>
>> When we switched to a tez-minimal and started getting errors while
>> launching containers, presumably because hadoop-* jars are not present in
>> the container launch classpath. How can this be fixed?
>>
>> Thanks,
>> Rajat
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/service/AbstractService
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.service.AbstractService
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> ... 13 more
>>
>
>
Re: Running using Tez minimal package
Posted by Siddharth Seth <ss...@apache.org>.
Setting tez.use.cluster.hadoop-libs to true is one option. This will try
picking up the classpath from the cluster itself - I believe a value
configured in yarn-site.xml. The drawback to this though is a that job
which are running during a rolling upgrade of the cluster can run into
issues - 1) Different tasks running with different versions of the
hadoop-client, 2) jars being replaced from under a running process.
Alternately, if you're trying to use a custom version of hadoop-libraries,
these can be place in HDFS and setup via 'tez.aux.uris'
The suggested deployment is with the regular package though. Is there a
specific reason for not making use of this. If a custom version of hadoop
is required, the package can be built by providing the -Dhadoop.version
flag to maven.
- Sid
On Tue, Feb 2, 2016 at 10:19 PM, Rajat Jain <ra...@gmail.com> wrote:
> In our current hive-on-tez deployment, we use the tez regular package
> which has hadoop-* jars in tez/lib.
>
> When we switched to a tez-minimal and started getting errors while
> launching containers, presumably because hadoop-* jars are not present in
> the container launch classpath. How can this be fixed?
>
> Thanks,
> Rajat
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/service/AbstractService
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.service.AbstractService
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 13 more
>