You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Vjeran Marcinko <vj...@email.t-com.hr> on 2013/06/18 07:46:37 UTC
Submitting jobs programmatically (even from Windows, IDE...)
Hi,
As a newcomer to Hadoop, I really struggled in the beginning to set things
up with running jobs from my IDE, and after I dove deep into how things
works, I discovered there are plenty of things that one has to take care,
and many parameters that can influence one's mind on how to best deploy the
jobs. Most of them I described in my blog with some additional utility code
that can help many to submit their jobs, together with any necessary 3rd
party libs:
<http://pannoniancoder.blogspot.com/2013/06/submitting-hadoop-jobs-programma
tically.html>
http://pannoniancoder.blogspot.com/2013/06/submitting-hadoop-jobs-programmat
ically.html
Moreover, I develop on Windows (don't ask me why), so one more thing to
worry about since Hadoop has not been developed with that OS in mind the
first place.
Regards,
Vjeran
RE: Submitting jobs programmatically (even from Windows, IDE...)
Posted by Chuan Liu <ch...@microsoft.com>.
Hi Vjeran,
First, thanks for the blog post!
As for the support for Windows, we are working actively to make Hadoop run smoothly on Windows.
The following JIRAs may provide some background and general outline of this effort.
https://issues.apache.org/jira/browse/HADOOP-8079
https://issues.apache.org/jira/browse/HADOOP-8562
https://issues.apache.org/jira/browse/MAPREDUCE-4401
https://issues.apache.org/jira/browse/YARN-191
You are very welcome to report any issues found on Windows as well.
Cheers,
Chuan
From: Vjeran Marcinko [mailto:vjeran.marcinko@email.t-com.hr]
Sent: Monday, June 17, 2013 10:47 PM
To: user@hadoop.apache.org
Subject: Submitting jobs programmatically (even from Windows, IDE...)
Hi,
As a newcomer to Hadoop, I really struggled in the beginning to set things up with running jobs from my IDE, and after I dove deep into how things works, I discovered there are plenty of things that one has to take care, and many parameters that can influence one's mind on how to best deploy the jobs. Most of them I described in my blog with some additional utility code that can help many to submit their jobs, together with any necessary 3rd party libs:
http://pannoniancoder.blogspot.com/2013/06/submitting-hadoop-jobs-programmatically.html
Moreover, I develop on Windows (don't ask me why), so one more thing to worry about since Hadoop has not been developed with that OS in mind the first place.
Regards,
Vjeran
RE: Submitting jobs programmatically (even from Windows, IDE...)
Posted by Chuan Liu <ch...@microsoft.com>.
Hi Vjeran,
First, thanks for the blog post!
As for the support for Windows, we are working actively to make Hadoop run smoothly on Windows.
The following JIRAs may provide some background and general outline of this effort.
https://issues.apache.org/jira/browse/HADOOP-8079
https://issues.apache.org/jira/browse/HADOOP-8562
https://issues.apache.org/jira/browse/MAPREDUCE-4401
https://issues.apache.org/jira/browse/YARN-191
You are very welcome to report any issues found on Windows as well.
Cheers,
Chuan
From: Vjeran Marcinko [mailto:vjeran.marcinko@email.t-com.hr]
Sent: Monday, June 17, 2013 10:47 PM
To: user@hadoop.apache.org
Subject: Submitting jobs programmatically (even from Windows, IDE...)
Hi,
As a newcomer to Hadoop, I really struggled in the beginning to set things up with running jobs from my IDE, and after I dove deep into how things works, I discovered there are plenty of things that one has to take care, and many parameters that can influence one's mind on how to best deploy the jobs. Most of them I described in my blog with some additional utility code that can help many to submit their jobs, together with any necessary 3rd party libs:
http://pannoniancoder.blogspot.com/2013/06/submitting-hadoop-jobs-programmatically.html
Moreover, I develop on Windows (don't ask me why), so one more thing to worry about since Hadoop has not been developed with that OS in mind the first place.
Regards,
Vjeran
RE: Submitting jobs programmatically (even from Windows, IDE...)
Posted by Chuan Liu <ch...@microsoft.com>.
Hi Vjeran,
First, thanks for the blog post!
As for the support for Windows, we are working actively to make Hadoop run smoothly on Windows.
The following JIRAs may provide some background and general outline of this effort.
https://issues.apache.org/jira/browse/HADOOP-8079
https://issues.apache.org/jira/browse/HADOOP-8562
https://issues.apache.org/jira/browse/MAPREDUCE-4401
https://issues.apache.org/jira/browse/YARN-191
You are very welcome to report any issues found on Windows as well.
Cheers,
Chuan
From: Vjeran Marcinko [mailto:vjeran.marcinko@email.t-com.hr]
Sent: Monday, June 17, 2013 10:47 PM
To: user@hadoop.apache.org
Subject: Submitting jobs programmatically (even from Windows, IDE...)
Hi,
As a newcomer to Hadoop, I really struggled in the beginning to set things up with running jobs from my IDE, and after I dove deep into how things works, I discovered there are plenty of things that one has to take care, and many parameters that can influence one's mind on how to best deploy the jobs. Most of them I described in my blog with some additional utility code that can help many to submit their jobs, together with any necessary 3rd party libs:
http://pannoniancoder.blogspot.com/2013/06/submitting-hadoop-jobs-programmatically.html
Moreover, I develop on Windows (don't ask me why), so one more thing to worry about since Hadoop has not been developed with that OS in mind the first place.
Regards,
Vjeran
RE: Submitting jobs programmatically (even from Windows, IDE...)
Posted by Chuan Liu <ch...@microsoft.com>.
Hi Vjeran,
First, thanks for the blog post!
As for the support for Windows, we are working actively to make Hadoop run smoothly on Windows.
The following JIRAs may provide some background and general outline of this effort.
https://issues.apache.org/jira/browse/HADOOP-8079
https://issues.apache.org/jira/browse/HADOOP-8562
https://issues.apache.org/jira/browse/MAPREDUCE-4401
https://issues.apache.org/jira/browse/YARN-191
You are very welcome to report any issues found on Windows as well.
Cheers,
Chuan
From: Vjeran Marcinko [mailto:vjeran.marcinko@email.t-com.hr]
Sent: Monday, June 17, 2013 10:47 PM
To: user@hadoop.apache.org
Subject: Submitting jobs programmatically (even from Windows, IDE...)
Hi,
As a newcomer to Hadoop, I really struggled in the beginning to set things up with running jobs from my IDE, and after I dove deep into how things works, I discovered there are plenty of things that one has to take care, and many parameters that can influence one's mind on how to best deploy the jobs. Most of them I described in my blog with some additional utility code that can help many to submit their jobs, together with any necessary 3rd party libs:
http://pannoniancoder.blogspot.com/2013/06/submitting-hadoop-jobs-programmatically.html
Moreover, I develop on Windows (don't ask me why), so one more thing to worry about since Hadoop has not been developed with that OS in mind the first place.
Regards,
Vjeran