You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by xeonmailinglist <xe...@gmail.com> on 2015/02/13 19:19:33 UTC
execute job in a remote jobtracker in YARN?
Hi,
I want to execute a job remotely. So, I was thinking in serialize the
org.apache.hadoop.mapreduce.Job class and send it to a remote component
that I create that launches the job there, or find a way to transform
the Job class into a configuration file that my remote component will
execute the job.
Is it possible to execute a Job in a remote jobtracker in YARN? If so,
what is the best way to do it?
Thanks,
WebHDFS and 100-continue
Posted by Ulul <ha...@ulul.org>.
Hi
Something is unclear to me in webHDFS API doc :
It states that the worklow is 2-steps (return 307 to redirect to
datanode, then client requests DN to upload data) as a workaround in
100-continue bug in Java/Jetty.
My understanding would be that in any case the redirect from NN to DN
would be kept and that we would have :
To NN: Expect: 100-continue
Response 307 DN
To DN : Expect: 100-continue
Response 100-continue
So the two steps would be kept
What do I miss there ?
Thanks for enlightment
Ulul
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File
WebHDFS and 100-continue
Posted by Ulul <ha...@ulul.org>.
Hi
Something is unclear to me in webHDFS API doc :
It states that the worklow is 2-steps (return 307 to redirect to
datanode, then client requests DN to upload data) as a workaround in
100-continue bug in Java/Jetty.
My understanding would be that in any case the redirect from NN to DN
would be kept and that we would have :
To NN: Expect: 100-continue
Response 307 DN
To DN : Expect: 100-continue
Response 100-continue
So the two steps would be kept
What do I miss there ?
Thanks for enlightment
Ulul
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File
Re: execute job in a remote jobtracker in YARN?
Posted by 杨浩 <ya...@gmail.com>.
Do you mean you want to execute a job in the remote cluster which don't
contain you node?
If you copy the configure of RM to your own computer, and this computer
will be taken as the hadoop client. Then you can execute the job through
'hadoop jar', the job will be executed on remote cluster
2015-02-14 6:48 GMT+08:00 Ravi Prakash <ra...@ymail.com>:
> Hi!
>
> There is no "JobTracker" in YARN. There is an ApplicationMaster. And there
> is a ResourceManager. Which do you mean?
>
> You can use the ResourceManager REST API to submit new applications
>
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
>
> Another option (and a rather convoluted one at that) is to create an MR
> job which retrieves the job jar and conf and parameters from a common
> source (FTP server e.g.) and launches a new job. This is something similar
> to what Oozie does. However its unlikely that you need to do the same thing.
>
> HTH
> Ravi
>
>
> On Friday, February 13, 2015 10:20 AM, xeonmailinglist <
> xeonmailinglist@gmail.com> wrote:
>
>
> Hi,
>
> I want to execute a job remotely. So, I was thinking in serialize the
> org.apache.hadoop.mapreduce.Job class and send it to a remote component
> that I create that launches the job there, or find a way to transform
> the Job class into a configuration file that my remote component will
> execute the job.
>
>
> Is it possible to execute a Job in a remote jobtracker in YARN? If so,
> what is the best way to do it?
>
> Thanks,
>
>
>
WebHDFS and 100-continue
Posted by Ulul <ha...@ulul.org>.
Hi
Something is unclear to me in webHDFS API doc :
It states that the worklow is 2-steps (return 307 to redirect to
datanode, then client requests DN to upload data) as a workaround in
100-continue bug in Java/Jetty.
My understanding would be that in any case the redirect from NN to DN
would be kept and that we would have :
To NN: Expect: 100-continue
Response 307 DN
To DN : Expect: 100-continue
Response 100-continue
So the two steps would be kept
What do I miss there ?
Thanks for enlightment
Ulul
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File
Re: execute job in a remote jobtracker in YARN?
Posted by 杨浩 <ya...@gmail.com>.
Do you mean you want to execute a job in the remote cluster which don't
contain you node?
If you copy the configure of RM to your own computer, and this computer
will be taken as the hadoop client. Then you can execute the job through
'hadoop jar', the job will be executed on remote cluster
2015-02-14 6:48 GMT+08:00 Ravi Prakash <ra...@ymail.com>:
> Hi!
>
> There is no "JobTracker" in YARN. There is an ApplicationMaster. And there
> is a ResourceManager. Which do you mean?
>
> You can use the ResourceManager REST API to submit new applications
>
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
>
> Another option (and a rather convoluted one at that) is to create an MR
> job which retrieves the job jar and conf and parameters from a common
> source (FTP server e.g.) and launches a new job. This is something similar
> to what Oozie does. However its unlikely that you need to do the same thing.
>
> HTH
> Ravi
>
>
> On Friday, February 13, 2015 10:20 AM, xeonmailinglist <
> xeonmailinglist@gmail.com> wrote:
>
>
> Hi,
>
> I want to execute a job remotely. So, I was thinking in serialize the
> org.apache.hadoop.mapreduce.Job class and send it to a remote component
> that I create that launches the job there, or find a way to transform
> the Job class into a configuration file that my remote component will
> execute the job.
>
>
> Is it possible to execute a Job in a remote jobtracker in YARN? If so,
> what is the best way to do it?
>
> Thanks,
>
>
>
WebHDFS and 100-continue
Posted by Ulul <ha...@ulul.org>.
Hi
Something is unclear to me in webHDFS API doc :
It states that the worklow is 2-steps (return 307 to redirect to
datanode, then client requests DN to upload data) as a workaround in
100-continue bug in Java/Jetty.
My understanding would be that in any case the redirect from NN to DN
would be kept and that we would have :
To NN: Expect: 100-continue
Response 307 DN
To DN : Expect: 100-continue
Response 100-continue
So the two steps would be kept
What do I miss there ?
Thanks for enlightment
Ulul
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File
Re: execute job in a remote jobtracker in YARN?
Posted by 杨浩 <ya...@gmail.com>.
Do you mean you want to execute a job in the remote cluster which don't
contain you node?
If you copy the configure of RM to your own computer, and this computer
will be taken as the hadoop client. Then you can execute the job through
'hadoop jar', the job will be executed on remote cluster
2015-02-14 6:48 GMT+08:00 Ravi Prakash <ra...@ymail.com>:
> Hi!
>
> There is no "JobTracker" in YARN. There is an ApplicationMaster. And there
> is a ResourceManager. Which do you mean?
>
> You can use the ResourceManager REST API to submit new applications
>
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
>
> Another option (and a rather convoluted one at that) is to create an MR
> job which retrieves the job jar and conf and parameters from a common
> source (FTP server e.g.) and launches a new job. This is something similar
> to what Oozie does. However its unlikely that you need to do the same thing.
>
> HTH
> Ravi
>
>
> On Friday, February 13, 2015 10:20 AM, xeonmailinglist <
> xeonmailinglist@gmail.com> wrote:
>
>
> Hi,
>
> I want to execute a job remotely. So, I was thinking in serialize the
> org.apache.hadoop.mapreduce.Job class and send it to a remote component
> that I create that launches the job there, or find a way to transform
> the Job class into a configuration file that my remote component will
> execute the job.
>
>
> Is it possible to execute a Job in a remote jobtracker in YARN? If so,
> what is the best way to do it?
>
> Thanks,
>
>
>
Re: execute job in a remote jobtracker in YARN?
Posted by 杨浩 <ya...@gmail.com>.
Do you mean you want to execute a job in the remote cluster which don't
contain you node?
If you copy the configure of RM to your own computer, and this computer
will be taken as the hadoop client. Then you can execute the job through
'hadoop jar', the job will be executed on remote cluster
2015-02-14 6:48 GMT+08:00 Ravi Prakash <ra...@ymail.com>:
> Hi!
>
> There is no "JobTracker" in YARN. There is an ApplicationMaster. And there
> is a ResourceManager. Which do you mean?
>
> You can use the ResourceManager REST API to submit new applications
>
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
>
> Another option (and a rather convoluted one at that) is to create an MR
> job which retrieves the job jar and conf and parameters from a common
> source (FTP server e.g.) and launches a new job. This is something similar
> to what Oozie does. However its unlikely that you need to do the same thing.
>
> HTH
> Ravi
>
>
> On Friday, February 13, 2015 10:20 AM, xeonmailinglist <
> xeonmailinglist@gmail.com> wrote:
>
>
> Hi,
>
> I want to execute a job remotely. So, I was thinking in serialize the
> org.apache.hadoop.mapreduce.Job class and send it to a remote component
> that I create that launches the job there, or find a way to transform
> the Job class into a configuration file that my remote component will
> execute the job.
>
>
> Is it possible to execute a Job in a remote jobtracker in YARN? If so,
> what is the best way to do it?
>
> Thanks,
>
>
>
Re: execute job in a remote jobtracker in YARN?
Posted by Ravi Prakash <ra...@ymail.com>.
Hi!
There is no "JobTracker" in YARN. There is an ApplicationMaster. And there is a ResourceManager. Which do you mean?
You can use the ResourceManager REST API to submit new applications
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
Another option (and a rather convoluted one at that) is to create an MR job which retrieves the job jar and conf and parameters from a common source (FTP server e.g.) and launches a new job. This is something similar to what Oozie does. However its unlikely that you need to do the same thing.
HTH
Ravi
On Friday, February 13, 2015 10:20 AM, xeonmailinglist <xe...@gmail.com> wrote:
Hi,
I want to execute a job remotely. So, I was thinking in serialize the
org.apache.hadoop.mapreduce.Job class and send it to a remote component
that I create that launches the job there, or find a way to transform
the Job class into a configuration file that my remote component will
execute the job.
Is it possible to execute a Job in a remote jobtracker in YARN? If so,
what is the best way to do it?
Thanks,
Re: execute job in a remote jobtracker in YARN?
Posted by Ravi Prakash <ra...@ymail.com>.
Hi!
There is no "JobTracker" in YARN. There is an ApplicationMaster. And there is a ResourceManager. Which do you mean?
You can use the ResourceManager REST API to submit new applications
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
Another option (and a rather convoluted one at that) is to create an MR job which retrieves the job jar and conf and parameters from a common source (FTP server e.g.) and launches a new job. This is something similar to what Oozie does. However its unlikely that you need to do the same thing.
HTH
Ravi
On Friday, February 13, 2015 10:20 AM, xeonmailinglist <xe...@gmail.com> wrote:
Hi,
I want to execute a job remotely. So, I was thinking in serialize the
org.apache.hadoop.mapreduce.Job class and send it to a remote component
that I create that launches the job there, or find a way to transform
the Job class into a configuration file that my remote component will
execute the job.
Is it possible to execute a Job in a remote jobtracker in YARN? If so,
what is the best way to do it?
Thanks,
Re: execute job in a remote jobtracker in YARN?
Posted by Ravi Prakash <ra...@ymail.com>.
Hi!
There is no "JobTracker" in YARN. There is an ApplicationMaster. And there is a ResourceManager. Which do you mean?
You can use the ResourceManager REST API to submit new applications
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
Another option (and a rather convoluted one at that) is to create an MR job which retrieves the job jar and conf and parameters from a common source (FTP server e.g.) and launches a new job. This is something similar to what Oozie does. However its unlikely that you need to do the same thing.
HTH
Ravi
On Friday, February 13, 2015 10:20 AM, xeonmailinglist <xe...@gmail.com> wrote:
Hi,
I want to execute a job remotely. So, I was thinking in serialize the
org.apache.hadoop.mapreduce.Job class and send it to a remote component
that I create that launches the job there, or find a way to transform
the Job class into a configuration file that my remote component will
execute the job.
Is it possible to execute a Job in a remote jobtracker in YARN? If so,
what is the best way to do it?
Thanks,
Re: execute job in a remote jobtracker in YARN?
Posted by Ravi Prakash <ra...@ymail.com>.
Hi!
There is no "JobTracker" in YARN. There is an ApplicationMaster. And there is a ResourceManager. Which do you mean?
You can use the ResourceManager REST API to submit new applications
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
Another option (and a rather convoluted one at that) is to create an MR job which retrieves the job jar and conf and parameters from a common source (FTP server e.g.) and launches a new job. This is something similar to what Oozie does. However its unlikely that you need to do the same thing.
HTH
Ravi
On Friday, February 13, 2015 10:20 AM, xeonmailinglist <xe...@gmail.com> wrote:
Hi,
I want to execute a job remotely. So, I was thinking in serialize the
org.apache.hadoop.mapreduce.Job class and send it to a remote component
that I create that launches the job there, or find a way to transform
the Job class into a configuration file that my remote component will
execute the job.
Is it possible to execute a Job in a remote jobtracker in YARN? If so,
what is the best way to do it?
Thanks,