Re: Can I submit jobs to a hadoop cluster without using hadoop executable

[vishalsant <Vi...@gmail.com>]

I opted ( for the time being ) unjar, combine in a single job jar.
It is not painful , till u have a large number of jars , some thing like
this 


 <target name="JOB" depends="clobber" >
	 	    <property name="a.jar" value="${dist.dir}/lib/a.jar" />
	 	    <property name="b.jar" value="${dist.dir}/lib/b.jar" />
	 	    <property name="extracted" value="${dist.dir}/extracted" />
	 	    <mkdir dir="${extracted}" />
	 	    <unjar dest="${extracted}" src="${a.jar}" />
	 	    <unjar dest="${extracted}" src="${b.jar}" />
	 	    <jar jarfile="${dist.job.dir}/JOB.jar"
		         basedir="${build.classes.dir}">
		    	<fileset dir="${build.classes.dir}">
		    			<include name="**/*.class" />
		 		</fileset>
		    	<fileset dir="${extracted}">
		    			<include name="**/*.class" />
		    	</fileset>
	 	      <manifest>
		        <attribute name="Main-Class" 
		                value="myClass"/>
		      </manifest>
		    </jar>
	 	<delete dir="${extracted}" />
   </target>


* Adding . if folks need to see one way.
 

vishalsant wrote:
> 
> I am a newbie to hadoop so please bear with me if this is naive.
> 
> I have defined a Mapper/Reducer and I desire to run it on a hadoop cluster
> My question is
> 
> * Do I need to specify the Mapper/Reducer in the classpath of all my
> DataNodes/JobTracker Node or can they be  uploaded to the cluster  as
> mobile code. 
> 
> I would like 
> 
> * to define , my job in it's totality on a client JVM ( independent of the
> cluster )
> * compile it 
> * run it's main method , with the conf pointing to the already
> established/running cluster.
> 
> The Mapper/Reducer should be serialized  to the JobTracker JVM , the
> classes deserialized  mapped to Mapper and Reducer and based on the input
> and output arguments map() and reduce() should execute. 
> 
> Is that even possible? 
> 
> Or do I have I have to manually move over to an hadoop installation and
> always execute it through the hadoop exec.
> 
> 

-- 
View this message in context: http://old.nabble.com/Can-I-submit-jobs-to-a-hadoop-cluster-without-using-hadoop-executable-tp27280041p28389503.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.