You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Evans Ye <ev...@apache.org> on 2015/05/20 18:42:28 UTC
Packaging and installing jars that needs to be available in HADOOP_CLASSPATH
Hi all,
I'm just thinking about the following problem and would like to get some
ideas from you experts.
I notice that the ignite-hadoop package needs to rely on puppet recipes to
symlik needed jars into /usr/lib/hadoop/lib, otherwise no mapreduce job
can be ran w/ ignite hadoop accelerator.
However, there might be a case that users are not using bigtop puppet, or,
sometimes users only need to quickly obtain an environment with specific
components installed and running. In that case, installing bigtop provided
package might be the easiest way. Take RPM packaging as an example, what do
you think if we can do symlink at the post-install phase of RPMs so that
the package can start working functionally right after it is installed?
Thanks,
Evans
Re: Packaging and installing jars that needs to be available in
HADOOP_CLASSPATH
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, Jun 10, 2015 at 02:39PM, Evans Ye wrote:
> Yes, that's what I was thinking. If we decide to leverage puppet recipes to
> complete the setup after packages are installed, then those packages can
> only be deployed by bigtop puppet instead of doing "yum install" or
> "apt-get install", which limited the usage of bigtop packages. It's fine to
> leave configurations for users but I think we better have symlink installed
> within package installation. :)
My only concern with this if that's an ok practice from the the linux
packaging guidelines point of view (if such think indeed exists). Otherwise, I
am for the simplification of the deployment.
Cos
> 2015-06-07 4:54 GMT+08:00 Olaf Flebbe <of...@oflebbe.de>:
>
> > Hi Evans,
> >
> > A +1 for creating symlinks in the post-install phase of the package,
> > since they are needed for the package to work, if I understand correctly.
> > packages itself should work (somehow) out of the box. Why are you not
> > considering packaging the symlinks directly in the RPM/DEB?
> >
> > Olaf
> >
> > > Am 20.05.2015 um 18:42 schrieb Evans Ye <ev...@apache.org>:
> > >
> > > Hi all,
> > >
> > > I'm just thinking about the following problem and would like to get some
> > > ideas from you experts.
> > >
> > > I notice that the ignite-hadoop package needs to rely on puppet recipes
> > to
> > > symlik needed jars into /usr/lib/hadoop/lib, otherwise no mapreduce job
> > > can be ran w/ ignite hadoop accelerator.
> > > However, there might be a case that users are not using bigtop puppet,
> > or,
> > > sometimes users only need to quickly obtain an environment with specific
> > > components installed and running. In that case, installing bigtop
> > provided
> > > package might be the easiest way. Take RPM packaging as an example, what
> > do
> > > you think if we can do symlink at the post-install phase of RPMs so that
> > > the package can start working functionally right after it is installed?
> > >
> > > Thanks,
> > > Evans
> >
> >
Re: Packaging and installing jars that needs to be available in HADOOP_CLASSPATH
Posted by Evans Ye <ev...@apache.org>.
Yes, that's what I was thinking. If we decide to leverage puppet recipes to
complete the setup after packages are installed, then those packages can
only be deployed by bigtop puppet instead of doing "yum install" or
"apt-get install", which limited the usage of bigtop packages. It's fine to
leave configurations for users but I think we better have symlink installed
within package installation. :)
2015-06-07 4:54 GMT+08:00 Olaf Flebbe <of...@oflebbe.de>:
> Hi Evans,
>
> A +1 for creating symlinks in the post-install phase of the package,
> since they are needed for the package to work, if I understand correctly.
> packages itself should work (somehow) out of the box. Why are you not
> considering packaging the symlinks directly in the RPM/DEB?
>
> Olaf
>
> > Am 20.05.2015 um 18:42 schrieb Evans Ye <ev...@apache.org>:
> >
> > Hi all,
> >
> > I'm just thinking about the following problem and would like to get some
> > ideas from you experts.
> >
> > I notice that the ignite-hadoop package needs to rely on puppet recipes
> to
> > symlik needed jars into /usr/lib/hadoop/lib, otherwise no mapreduce job
> > can be ran w/ ignite hadoop accelerator.
> > However, there might be a case that users are not using bigtop puppet,
> or,
> > sometimes users only need to quickly obtain an environment with specific
> > components installed and running. In that case, installing bigtop
> provided
> > package might be the easiest way. Take RPM packaging as an example, what
> do
> > you think if we can do symlink at the post-install phase of RPMs so that
> > the package can start working functionally right after it is installed?
> >
> > Thanks,
> > Evans
>
>
Re: Packaging and installing jars that needs to be available in HADOOP_CLASSPATH
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi Evans,
A +1 for creating symlinks in the post-install phase of the package, since they are needed for the package to work, if I understand correctly. packages itself should work (somehow) out of the box. Why are you not considering packaging the symlinks directly in the RPM/DEB?
Olaf
> Am 20.05.2015 um 18:42 schrieb Evans Ye <ev...@apache.org>:
>
> Hi all,
>
> I'm just thinking about the following problem and would like to get some
> ideas from you experts.
>
> I notice that the ignite-hadoop package needs to rely on puppet recipes to
> symlik needed jars into /usr/lib/hadoop/lib, otherwise no mapreduce job
> can be ran w/ ignite hadoop accelerator.
> However, there might be a case that users are not using bigtop puppet, or,
> sometimes users only need to quickly obtain an environment with specific
> components installed and running. In that case, installing bigtop provided
> package might be the easiest way. Take RPM packaging as an example, what do
> you think if we can do symlink at the post-install phase of RPMs so that
> the package can start working functionally right after it is installed?
>
> Thanks,
> Evans
Re: Packaging and installing jars that needs to be available in
HADOOP_CLASSPATH
Posted by Konstantin Boudnik <co...@apache.org>.
Guys,
the reason I put this logic to the Puppet is because I felt uncomfortable of
adding things to the folder belonging to a different package. If you think
this is an acceptable practice we surely can move to the ignite-hadoop package
from Puppet recipes.
Thanks,
Cos
On Fri, May 22, 2015 at 02:33PM, Evans Ye wrote:
> Hi mark,
>
> Thanks for your answer.
> I also do not know to many about ignite.
> But moving the ignite jars out of HADOOP_CLASSPATH cause
> ClassNotFoundException when accessing IGFS
>
> [root@bigtop1 lib]# hadoop --config /etc/hadoop/ignite.client.conf/ fs -ls /
> -ls: Fatal internal error
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem not found
> at
> ...
>
> So put it under /usr/lib/hadoop/lib seems to be necessary.
>
> So, back to the packaging problem, follow your logic I think the best way
> to do address the problem is to install ignite jars symlinks at the same
> time when iginte-hadoop rpm is installed. i.e. we can do this in post
> install section of rpm.
>
>
> 2015-05-21 0:51 GMT+08:00 Mark Grover <ma...@apache.org>:
>
> > Evans,
> > The rule of thumb there usually is to create an extra package that depends
> > on hadoop in this case and delivers jars under /usr/lib/hadoop/lib.
> >
> > The dependency on hadoop makes sure that the directory /usr/lib/hadoop/lib
> > already exists because this new package is installed. So, your final users
> > will only have to install this new package in order to get hadoop and the
> > ignite dependencies in the hadoop classpath. It also allows people who
> > don't want ignite in their classpath to simply install hadoop not have it
> > in their classpath.
> >
> > BTW, I admit to not knowing everything about ignite, but if MR classpath is
> > all you want it in, it may make sense to put the jars under
> > /usr/lib/hadoop-mapreduce (and depend on the hadoop-mapreduce package
> > instaead of the hadoop package). No need to have ignite jars in the HDFS
> > classpath which will be the case if you were to put them under
> > /usr/lib/hadoop, as far I as recall.
> >
> > Mark
> >
> > On Wed, May 20, 2015 at 9:42 AM, Evans Ye <ev...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > I'm just thinking about the following problem and would like to get some
> > > ideas from you experts.
> > >
> > > I notice that the ignite-hadoop package needs to rely on puppet recipes
> > to
> > > symlik needed jars into /usr/lib/hadoop/lib, otherwise no mapreduce job
> > > can be ran w/ ignite hadoop accelerator.
> > > However, there might be a case that users are not using bigtop puppet,
> > or,
> > > sometimes users only need to quickly obtain an environment with specific
> > > components installed and running. In that case, installing bigtop
> > provided
> > > package might be the easiest way. Take RPM packaging as an example, what
> > do
> > > you think if we can do symlink at the post-install phase of RPMs so that
> > > the package can start working functionally right after it is installed?
> > >
> > > Thanks,
> > > Evans
> > >
> >
Re: Packaging and installing jars that needs to be available in HADOOP_CLASSPATH
Posted by Evans Ye <ev...@apache.org>.
Hi mark,
Thanks for your answer.
I also do not know to many about ignite.
But moving the ignite jars out of HADOOP_CLASSPATH cause
ClassNotFoundException when accessing IGFS
[root@bigtop1 lib]# hadoop --config /etc/hadoop/ignite.client.conf/ fs -ls /
-ls: Fatal internal error
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem not found
at
...
So put it under /usr/lib/hadoop/lib seems to be necessary.
So, back to the packaging problem, follow your logic I think the best way
to do address the problem is to install ignite jars symlinks at the same
time when iginte-hadoop rpm is installed. i.e. we can do this in post
install section of rpm.
2015-05-21 0:51 GMT+08:00 Mark Grover <ma...@apache.org>:
> Evans,
> The rule of thumb there usually is to create an extra package that depends
> on hadoop in this case and delivers jars under /usr/lib/hadoop/lib.
>
> The dependency on hadoop makes sure that the directory /usr/lib/hadoop/lib
> already exists because this new package is installed. So, your final users
> will only have to install this new package in order to get hadoop and the
> ignite dependencies in the hadoop classpath. It also allows people who
> don't want ignite in their classpath to simply install hadoop not have it
> in their classpath.
>
> BTW, I admit to not knowing everything about ignite, but if MR classpath is
> all you want it in, it may make sense to put the jars under
> /usr/lib/hadoop-mapreduce (and depend on the hadoop-mapreduce package
> instaead of the hadoop package). No need to have ignite jars in the HDFS
> classpath which will be the case if you were to put them under
> /usr/lib/hadoop, as far I as recall.
>
> Mark
>
> On Wed, May 20, 2015 at 9:42 AM, Evans Ye <ev...@apache.org> wrote:
>
> > Hi all,
> >
> > I'm just thinking about the following problem and would like to get some
> > ideas from you experts.
> >
> > I notice that the ignite-hadoop package needs to rely on puppet recipes
> to
> > symlik needed jars into /usr/lib/hadoop/lib, otherwise no mapreduce job
> > can be ran w/ ignite hadoop accelerator.
> > However, there might be a case that users are not using bigtop puppet,
> or,
> > sometimes users only need to quickly obtain an environment with specific
> > components installed and running. In that case, installing bigtop
> provided
> > package might be the easiest way. Take RPM packaging as an example, what
> do
> > you think if we can do symlink at the post-install phase of RPMs so that
> > the package can start working functionally right after it is installed?
> >
> > Thanks,
> > Evans
> >
>
Re: Packaging and installing jars that needs to be available in HADOOP_CLASSPATH
Posted by Mark Grover <ma...@apache.org>.
Evans,
The rule of thumb there usually is to create an extra package that depends
on hadoop in this case and delivers jars under /usr/lib/hadoop/lib.
The dependency on hadoop makes sure that the directory /usr/lib/hadoop/lib
already exists because this new package is installed. So, your final users
will only have to install this new package in order to get hadoop and the
ignite dependencies in the hadoop classpath. It also allows people who
don't want ignite in their classpath to simply install hadoop not have it
in their classpath.
BTW, I admit to not knowing everything about ignite, but if MR classpath is
all you want it in, it may make sense to put the jars under
/usr/lib/hadoop-mapreduce (and depend on the hadoop-mapreduce package
instaead of the hadoop package). No need to have ignite jars in the HDFS
classpath which will be the case if you were to put them under
/usr/lib/hadoop, as far I as recall.
Mark
On Wed, May 20, 2015 at 9:42 AM, Evans Ye <ev...@apache.org> wrote:
> Hi all,
>
> I'm just thinking about the following problem and would like to get some
> ideas from you experts.
>
> I notice that the ignite-hadoop package needs to rely on puppet recipes to
> symlik needed jars into /usr/lib/hadoop/lib, otherwise no mapreduce job
> can be ran w/ ignite hadoop accelerator.
> However, there might be a case that users are not using bigtop puppet, or,
> sometimes users only need to quickly obtain an environment with specific
> components installed and running. In that case, installing bigtop provided
> package might be the easiest way. Take RPM packaging as an example, what do
> you think if we can do symlink at the post-install phase of RPMs so that
> the package can start working functionally right after it is installed?
>
> Thanks,
> Evans
>