Re: Hadoop on Solaris 10 and in OSGi bundles

[kovachev <de...@yahoo.com>]

Hi,

we are trying to set up Hadoop to run on Solaris 10 within Containers.
However, we encounter many problems. 
Could you please write here down all the extra settings needed for running
Hadoop on Solaris?

Tnx,
Dejan


Jon Brisbin-2 wrote:
> 
> Has anyone got Hadoop and/or HBase to OSGify? I Googled for a half- 
> hour on this topic and there's almost no mention of OSGi and Hadoop  
> (one JIRA ticket that's more a tutorial on writing Hadoop patches than  
> information about the problem). There are plenty of issues with  
> classloading, I'm sure, but at the moment, I'm not running remote  
> cluster tasks inside OSGi, just trying to write files into HDFS and  
> HBase and submit jobs. I can't get past this first problem, though, so  
> I haven't been able to make much progress.
> 
> I'm trying to evaluate whether SpringSource dm Server will work for us  
> to replace JBoss. I was planning on implementing a Hadoop cluster for  
> back-end services rather than trying to get more servers to do load- 
> balancing (the work for various pages is getting bigger and bigger all  
> the time). I deploy in Java 6 on Solaris 10 x86.
> 
> I was able to get plain-jane Hadoop 0.18.1 running on Solaris 10 after  
> overriding all the 0.0.0.0 IP addresses in the default file with $ 
> {hadoop.bind.address} and specifying -Dhadoop.bind.address=1.2.3.4 in  
> hadoop-env.sh (it was throwing NPEs before I did that). On Solaris 10,  
> whoami is in /usr/ucb, which isn't in the PATH by default, so the docs  
> should say you need to add /usr/ucb to the the path if you're using  
> Solaris (which I'm assuming not many people are doing since there's  
> ZERO documentation for the platform).
> 
> I couldn't get the native libraries to compile at all (make complains  
> about macros because the Solaris gmake is just a shill for BSD make  
> in /usr/ccs/bin).
> 
> +-( /opt/hadoop-0.18.1/src/native ):# gmake
> make  all-recursive
> make: Fatal error in reader: Makefile, line 647: Badly formed macro  
> assignment
> Current working directory /opt/hadoop-0.18.1/src/native
> gmake: *** [all] Error 1
> 
> Forget about using the Sun Studio compiler.
> 
> But the biggest bummer was trying to get bnd (
> http://www.aqute.biz/Code/Bnd 
>   ) to create an OSGi bundle from Hadoop (which I was able to do with  
> ZooKeeper...clients are able to connect to ZooKeeper server fine in a  
> Spring app on SpringSource dm/Equinox). I was just wanting to use the  
> client functionality from the OSGi container, not embed a server. bnd  
> gets confused with the "webapps" folder (among other things). I got it  
> to ignore that for Import-Package and Export-Package headers, but  
> Equinox is complaining that the manifest is screwed up. Bottom line  
> seems to be that I can't use bnd to create an OSGi bundle for Hadoop  
> 0.18.1 like I did with ZooKeeper. Maybe I can create new bundles using  
> eclilpse PDE, then import the source code, let eclipse compile  
> everything (cross fingers that most dependencies are either already  
> OSGified or can be OSGified with bnd), then export an OSGi bundle for  
> the various pieces. Not sure if anyone's tried this yet.
> 
> A SpringSource-compatible bundle would put webapps in a bundle with a  
> "web" personality and ideally there'd be a segregation of packages  
> into client and server bundles (there looks to be other packages in  
> there which would probably have to go into their own bundle, as well).  
> In fact, it would be awesome if the whole thing could be dropped into  
> Equinox and embedded into a Spring application. At the very least I'd  
> like to be able to connect to HDFS without resorting to stuffing the  
> Hadoop jar into my bundle. Extra-special functionality would include  
> submitting Hadoop jobs and reading HBase tables and the like. Extra- 
> extra special functionality would be deploying jobs and tasks inside  
> Equinox on the cluster.
> 
> Thanks!
> 
> Jon Brisbin
> Portal Webmaster
> NPC International, Inc.
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Hadoop-on-Solaris-10-and-in-OSGi-bundles-tp20361820p28391003.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.

Re: Hadoop on Solaris 10 and in OSGi bundles

[Allen Wittenauer <aw...@linkedin.com>]

On Apr 28, 2010, at 9:30 AM, kovachev wrote:
> we are trying to set up Hadoop to run on Solaris 10 within Containers.
> However, we encounter many problems. 
> Could you please write here down all the extra settings needed for running
> Hadoop on Solaris?

The two big ones:

	- whoami needs to be in the path because Hadoop is too stupid to figure out who is using it without a shell out
	- For HADOOP_IDENT_STRING we use `/usr/xpg4/bin/id -u -n`

Also note that Hadoop can get confused about pooled storage.  In the case of ZFS, creating FS's such as:

u007                   914G    24K   345G     1%    /mnt/u007
u007/mapred            200G   7.2G   193G     4%    /mnt/u007/mapred
u007/dfs-data          914G   369G   345G    52%    /mnt/u007/dfs-data

will cause HDFS to miscalculate the total capacity of a datanode and the entire cluster.  So just remember there is less storage there than what it tells you.  [I think there is a JIRA on this somewhere, but because so few of us use pooled storage, it is unlikely to get fixed.]

	- If you use the Cloudera dist, you'll likely need to back some of their changes out in order for it to work properly.  [We don't anymore, so dunno if that is still the case.]