You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Bruce Williams <wi...@gmail.com> on 2010/10/16 22:08:45 UTC
Hadoop - Solaris
Hi,
I am doing a student Independent Study Project and Harvery Mudd has given
me 13 Sun Netra X1 I can use as a dedicated Hadoop cluster. Right now they
are without an OS.
If anyone with experience with Hadoop and Solaris can contact me off list,
even to just say I am doing it and it is OK it would be appreciated.
Thanks in advance,
Bruce Williams
-----------------------------------------------------------------------------------------------------------------------------
Re: Hadoop - Solaris
Posted by Bruce Williams <wi...@gmail.com>.
I didn't want to jump in and say thanks for each post but thanks for
all. Owen, you probably don't remember me, but we met at that first
Hadoop meetup in Sunnyvale a couple years ago.
Bruce Williams
-----------------------------------------------------------------------------------------------------------------------------
On Sun, Oct 17, 2010 at 5:44 PM, Allen Wittenauer
<aw...@linkedin.com> wrote:
>
> On Oct 17, 2010, at 2:41 PM, Gregor Willemsen wrote:
>
>> Of course it is simple if you know about it. From an Linux and Java
>> point of view the Solaris way is not that obvious.
>
> Java has nothing to do with GNU tar's extensions or gcc.
>
> Forward compatibility is definitely not the Linux way, so it is easy to understand the confusion.
Re: Hadoop - Solaris
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Oct 17, 2010, at 2:41 PM, Gregor Willemsen wrote:
> Of course it is simple if you know about it. From an Linux and Java
> point of view the Solaris way is not that obvious.
Java has nothing to do with GNU tar's extensions or gcc.
Forward compatibility is definitely not the Linux way, so it is easy to understand the confusion.
Re: Hadoop - Solaris
Posted by Gregor Willemsen <gr...@googlemail.com>.
Of course it is simple if you know about it. From an Linux and Java
point of view the Solaris way is not that obvious.
My 2 cents
2010/10/17 Allen Wittenauer <aw...@linkedin.com>:
>
> On Oct 17, 2010, at 3:18 AM, Gregor Willemsen wrote:
>
>> In
>> 2007 or 2008 GNU and Solaris did not play together very well as things
>> as tar, gcc are concerned.
>
> Only if you didn't know to look in /usr/sfw or were using such an ancient version of Solaris that one be embarrassed.
>
>
>
Re: Hadoop - Solaris
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Oct 17, 2010, at 3:18 AM, Gregor Willemsen wrote:
> In
> 2007 or 2008 GNU and Solaris did not play together very well as things
> as tar, gcc are concerned.
Only if you didn't know to look in /usr/sfw or were using such an ancient version of Solaris that one be embarrassed.
Re: Hadoop - Solaris
Posted by Gregor Willemsen <gr...@googlemail.com>.
I have no recent experiences with Solaris nor with Hadoop on Solaris,
but there is an blog: http://blogs.sun.com/george/category/Hadoop with
three postings dedicated to Hadoop (although maybe a bit outdated). In
2007 or 2008 GNU and Solaris did not play together very well as things
as tar, gcc are concerned. But these issues should have gone now.
Sun Grid Engine might be a starting point to look for as Hadoop is built in.
2010/10/16 Allen Wittenauer <aw...@linkedin.com>:
>
> On Oct 16, 2010, at 1:08 PM, Bruce Williams wrote:
>> I am doing a student Independent Study Project and Harvery Mudd has given
>> me 13 Sun Netra X1 I can use as a dedicated Hadoop cluster. Right now they
>> are without an OS.
>>
>> If anyone with experience with Hadoop and Solaris can contact me off list,
>> even to just say I am doing it and it is OK it would be appreciated.
>
>
>
> That's my cue! :)
>
> We have a few grids that are running Solaris. It mostly works out of the box as long as you are aware of three things:
>
> - There are some settings in hadoop-env.sh and in the path that need to be dealt with. Rather than re-quote, these were added to the Hadoop FAQ a week or two ago so definitely take a look at that.
>
> - The native compression libraries will need to be compiled. Depending upon what you are doing/how performant the machines are, this may or may not make a big difference. Compiling under Solaris with gcc will work fine (but it is gcc... ugh!). Only a few minor changes are required to compile it with SUNWspro. [I have patches laying around here somewhere if anyone wants to play with them.]
>
> - The Solaris JRE is a mixed-mode implementation. So keep in mind that -d32 and -d64 have meaning and do work as advertised. You'll likely want to pick a bitsize and use that for all your Hadoop daemons and tasks, especially if you plan on using any JNI like the compression libraries.
Re: Hadoop - Solaris
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Oct 16, 2010, at 1:08 PM, Bruce Williams wrote:
> I am doing a student Independent Study Project and Harvery Mudd has given
> me 13 Sun Netra X1 I can use as a dedicated Hadoop cluster. Right now they
> are without an OS.
>
> If anyone with experience with Hadoop and Solaris can contact me off list,
> even to just say I am doing it and it is OK it would be appreciated.
That's my cue! :)
We have a few grids that are running Solaris. It mostly works out of the box as long as you are aware of three things:
- There are some settings in hadoop-env.sh and in the path that need to be dealt with. Rather than re-quote, these were added to the Hadoop FAQ a week or two ago so definitely take a look at that.
- The native compression libraries will need to be compiled. Depending upon what you are doing/how performant the machines are, this may or may not make a big difference. Compiling under Solaris with gcc will work fine (but it is gcc... ugh!). Only a few minor changes are required to compile it with SUNWspro. [I have patches laying around here somewhere if anyone wants to play with them.]
- The Solaris JRE is a mixed-mode implementation. So keep in mind that -d32 and -d64 have meaning and do work as advertised. You'll likely want to pick a bitsize and use that for all your Hadoop daemons and tasks, especially if you plan on using any JNI like the compression libraries.
Re: Hadoop - Solaris
Posted by Owen O'Malley <om...@apache.org>.
On Oct 16, 2010, at 1:08 PM, Bruce Williams wrote:
> If anyone with experience with Hadoop and Solaris can contact me off
> list,
> even to just say I am doing it and it is OK it would be appreciated.
LinkedIn is currently running Hadoop on Solaris. Hopefully, Allen
Wittenauer can get back to you on some hints.
-- Owen