You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Answer Agrawal <yr...@gmail.com> on 2015/04/11 08:23:45 UTC
Can we run mapreduce job from eclipse IDE on fully distributed mode
hadoop cluster?
A mapreduce job can be run as jar file from terminal or directly from
eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
and all resources of cluster. Does the same thing happen when we run from
IDE. I have run a job on both and it takes less time on IDE than jar file
on terminal.
Thanks
Re: Can we run mapreduce job from eclipse IDE on fully distributed
mode hadoop cluster?
Posted by 杨浩 <ya...@gmail.com>.
I think you can have a try at http://hdt.incubator.apache.org/
2015-04-12 2:04 GMT+08:00 Answer Agrawal <yr...@gmail.com>:
> Thanks Jonathan
>
> I have installed and configured my own hadoop cluster with one master node
> and 7 slave nodes. Now I just want to make sure that job running through
> Eclipse is internally same as running through jar file. Also job history
> server and web ui 8088 shows only list of those jobs which are submitted
> through jar using terminal.
>
>
>
>
> On Sat, Apr 11, 2015 at 12:25 PM, Jonathan Aquilina <
> jaquilina@eagleeyet.net> wrote:
>
>> I could be wrong here, but the way I understand things I do not think
>> that is even possible to run the JAR file from your PC. There are two
>> things that you need to realize.
>>
>> 1) How is the JAR file going to connect to the cluster
>>
>> 2) How is the JAR file going to be distributed to the cluster.
>>
>> Again I could be wrong here in my response, so anyone else on the list
>> feel free to correct me. I am still a novice to Hadoop and have only worked
>> with it on amazon EMR.
>>
>> ---
>> Regards,
>> Jonathan Aquilina
>> Founder Eagle Eye T
>>
>> On 2015-04-11 08:23, Answer Agrawal wrote:
>>
>> A mapreduce job can be run as jar file from terminal or directly from
>> eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
>> and all resources of cluster. Does the same thing happen when we run from
>> IDE. I have run a job on both and it takes less time on IDE than jar file
>> on terminal.
>>
>> Thanks
>>
>>
>>
>
Re: Can we run mapreduce job from eclipse IDE on fully distributed
mode hadoop cluster?
Posted by 杨浩 <ya...@gmail.com>.
I think you can have a try at http://hdt.incubator.apache.org/
2015-04-12 2:04 GMT+08:00 Answer Agrawal <yr...@gmail.com>:
> Thanks Jonathan
>
> I have installed and configured my own hadoop cluster with one master node
> and 7 slave nodes. Now I just want to make sure that job running through
> Eclipse is internally same as running through jar file. Also job history
> server and web ui 8088 shows only list of those jobs which are submitted
> through jar using terminal.
>
>
>
>
> On Sat, Apr 11, 2015 at 12:25 PM, Jonathan Aquilina <
> jaquilina@eagleeyet.net> wrote:
>
>> I could be wrong here, but the way I understand things I do not think
>> that is even possible to run the JAR file from your PC. There are two
>> things that you need to realize.
>>
>> 1) How is the JAR file going to connect to the cluster
>>
>> 2) How is the JAR file going to be distributed to the cluster.
>>
>> Again I could be wrong here in my response, so anyone else on the list
>> feel free to correct me. I am still a novice to Hadoop and have only worked
>> with it on amazon EMR.
>>
>> ---
>> Regards,
>> Jonathan Aquilina
>> Founder Eagle Eye T
>>
>> On 2015-04-11 08:23, Answer Agrawal wrote:
>>
>> A mapreduce job can be run as jar file from terminal or directly from
>> eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
>> and all resources of cluster. Does the same thing happen when we run from
>> IDE. I have run a job on both and it takes less time on IDE than jar file
>> on terminal.
>>
>> Thanks
>>
>>
>>
>
Re: Can we run mapreduce job from eclipse IDE on fully distributed
mode hadoop cluster?
Posted by 杨浩 <ya...@gmail.com>.
I think you can have a try at http://hdt.incubator.apache.org/
2015-04-12 2:04 GMT+08:00 Answer Agrawal <yr...@gmail.com>:
> Thanks Jonathan
>
> I have installed and configured my own hadoop cluster with one master node
> and 7 slave nodes. Now I just want to make sure that job running through
> Eclipse is internally same as running through jar file. Also job history
> server and web ui 8088 shows only list of those jobs which are submitted
> through jar using terminal.
>
>
>
>
> On Sat, Apr 11, 2015 at 12:25 PM, Jonathan Aquilina <
> jaquilina@eagleeyet.net> wrote:
>
>> I could be wrong here, but the way I understand things I do not think
>> that is even possible to run the JAR file from your PC. There are two
>> things that you need to realize.
>>
>> 1) How is the JAR file going to connect to the cluster
>>
>> 2) How is the JAR file going to be distributed to the cluster.
>>
>> Again I could be wrong here in my response, so anyone else on the list
>> feel free to correct me. I am still a novice to Hadoop and have only worked
>> with it on amazon EMR.
>>
>> ---
>> Regards,
>> Jonathan Aquilina
>> Founder Eagle Eye T
>>
>> On 2015-04-11 08:23, Answer Agrawal wrote:
>>
>> A mapreduce job can be run as jar file from terminal or directly from
>> eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
>> and all resources of cluster. Does the same thing happen when we run from
>> IDE. I have run a job on both and it takes less time on IDE than jar file
>> on terminal.
>>
>> Thanks
>>
>>
>>
>
Re: Can we run mapreduce job from eclipse IDE on fully distributed
mode hadoop cluster?
Posted by 杨浩 <ya...@gmail.com>.
I think you can have a try at http://hdt.incubator.apache.org/
2015-04-12 2:04 GMT+08:00 Answer Agrawal <yr...@gmail.com>:
> Thanks Jonathan
>
> I have installed and configured my own hadoop cluster with one master node
> and 7 slave nodes. Now I just want to make sure that job running through
> Eclipse is internally same as running through jar file. Also job history
> server and web ui 8088 shows only list of those jobs which are submitted
> through jar using terminal.
>
>
>
>
> On Sat, Apr 11, 2015 at 12:25 PM, Jonathan Aquilina <
> jaquilina@eagleeyet.net> wrote:
>
>> I could be wrong here, but the way I understand things I do not think
>> that is even possible to run the JAR file from your PC. There are two
>> things that you need to realize.
>>
>> 1) How is the JAR file going to connect to the cluster
>>
>> 2) How is the JAR file going to be distributed to the cluster.
>>
>> Again I could be wrong here in my response, so anyone else on the list
>> feel free to correct me. I am still a novice to Hadoop and have only worked
>> with it on amazon EMR.
>>
>> ---
>> Regards,
>> Jonathan Aquilina
>> Founder Eagle Eye T
>>
>> On 2015-04-11 08:23, Answer Agrawal wrote:
>>
>> A mapreduce job can be run as jar file from terminal or directly from
>> eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
>> and all resources of cluster. Does the same thing happen when we run from
>> IDE. I have run a job on both and it takes less time on IDE than jar file
>> on terminal.
>>
>> Thanks
>>
>>
>>
>
Re: Can we run mapreduce job from eclipse IDE on fully distributed
mode hadoop cluster?
Posted by Answer Agrawal <yr...@gmail.com>.
Thanks Jonathan
I have installed and configured my own hadoop cluster with one master node
and 7 slave nodes. Now I just want to make sure that job running through
Eclipse is internally same as running through jar file. Also job history
server and web ui 8088 shows only list of those jobs which are submitted
through jar using terminal.
On Sat, Apr 11, 2015 at 12:25 PM, Jonathan Aquilina <jaquilina@eagleeyet.net
> wrote:
> I could be wrong here, but the way I understand things I do not think
> that is even possible to run the JAR file from your PC. There are two
> things that you need to realize.
>
> 1) How is the JAR file going to connect to the cluster
>
> 2) How is the JAR file going to be distributed to the cluster.
>
> Again I could be wrong here in my response, so anyone else on the list
> feel free to correct me. I am still a novice to Hadoop and have only worked
> with it on amazon EMR.
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-04-11 08:23, Answer Agrawal wrote:
>
> A mapreduce job can be run as jar file from terminal or directly from
> eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
> and all resources of cluster. Does the same thing happen when we run from
> IDE. I have run a job on both and it takes less time on IDE than jar file
> on terminal.
>
> Thanks
>
>
>
Re: Can we run mapreduce job from eclipse IDE on fully distributed
mode hadoop cluster?
Posted by Answer Agrawal <yr...@gmail.com>.
Thanks Jonathan
I have installed and configured my own hadoop cluster with one master node
and 7 slave nodes. Now I just want to make sure that job running through
Eclipse is internally same as running through jar file. Also job history
server and web ui 8088 shows only list of those jobs which are submitted
through jar using terminal.
On Sat, Apr 11, 2015 at 12:25 PM, Jonathan Aquilina <jaquilina@eagleeyet.net
> wrote:
> I could be wrong here, but the way I understand things I do not think
> that is even possible to run the JAR file from your PC. There are two
> things that you need to realize.
>
> 1) How is the JAR file going to connect to the cluster
>
> 2) How is the JAR file going to be distributed to the cluster.
>
> Again I could be wrong here in my response, so anyone else on the list
> feel free to correct me. I am still a novice to Hadoop and have only worked
> with it on amazon EMR.
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-04-11 08:23, Answer Agrawal wrote:
>
> A mapreduce job can be run as jar file from terminal or directly from
> eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
> and all resources of cluster. Does the same thing happen when we run from
> IDE. I have run a job on both and it takes less time on IDE than jar file
> on terminal.
>
> Thanks
>
>
>
Re: Can we run mapreduce job from eclipse IDE on fully distributed
mode hadoop cluster?
Posted by Answer Agrawal <yr...@gmail.com>.
Thanks Jonathan
I have installed and configured my own hadoop cluster with one master node
and 7 slave nodes. Now I just want to make sure that job running through
Eclipse is internally same as running through jar file. Also job history
server and web ui 8088 shows only list of those jobs which are submitted
through jar using terminal.
On Sat, Apr 11, 2015 at 12:25 PM, Jonathan Aquilina <jaquilina@eagleeyet.net
> wrote:
> I could be wrong here, but the way I understand things I do not think
> that is even possible to run the JAR file from your PC. There are two
> things that you need to realize.
>
> 1) How is the JAR file going to connect to the cluster
>
> 2) How is the JAR file going to be distributed to the cluster.
>
> Again I could be wrong here in my response, so anyone else on the list
> feel free to correct me. I am still a novice to Hadoop and have only worked
> with it on amazon EMR.
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-04-11 08:23, Answer Agrawal wrote:
>
> A mapreduce job can be run as jar file from terminal or directly from
> eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
> and all resources of cluster. Does the same thing happen when we run from
> IDE. I have run a job on both and it takes less time on IDE than jar file
> on terminal.
>
> Thanks
>
>
>
Re: Can we run mapreduce job from eclipse IDE on fully distributed
mode hadoop cluster?
Posted by Answer Agrawal <yr...@gmail.com>.
Thanks Jonathan
I have installed and configured my own hadoop cluster with one master node
and 7 slave nodes. Now I just want to make sure that job running through
Eclipse is internally same as running through jar file. Also job history
server and web ui 8088 shows only list of those jobs which are submitted
through jar using terminal.
On Sat, Apr 11, 2015 at 12:25 PM, Jonathan Aquilina <jaquilina@eagleeyet.net
> wrote:
> I could be wrong here, but the way I understand things I do not think
> that is even possible to run the JAR file from your PC. There are two
> things that you need to realize.
>
> 1) How is the JAR file going to connect to the cluster
>
> 2) How is the JAR file going to be distributed to the cluster.
>
> Again I could be wrong here in my response, so anyone else on the list
> feel free to correct me. I am still a novice to Hadoop and have only worked
> with it on amazon EMR.
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-04-11 08:23, Answer Agrawal wrote:
>
> A mapreduce job can be run as jar file from terminal or directly from
> eclipse IDE. When a job run as jar file from terminal it uses multiple jvm
> and all resources of cluster. Does the same thing happen when we run from
> IDE. I have run a job on both and it takes less time on IDE than jar file
> on terminal.
>
> Thanks
>
>
>
Re: Can we run mapreduce job from eclipse IDE on fully distributed mode hadoop cluster?
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
I could be wrong here, but the way I understand things I do not think
that is even possible to run the JAR file from your PC. There are two
things that you need to realize.
1) How is the JAR file going to connect to the cluster
2) How is the JAR file going to be distributed to the cluster.
Again I could be wrong here in my response, so anyone else on the list
feel free to correct me. I am still a novice to Hadoop and have only
worked with it on amazon EMR.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-04-11 08:23, Answer Agrawal wrote:
> A mapreduce job can be run as jar file from terminal or directly from eclipse IDE. When a job run as jar file from terminal it uses multiple jvm and all resources of cluster. Does the same thing happen when we run from IDE. I have run a job on both and it takes less time on IDE than jar file on terminal.
>
> Thanks
Re: Can we run mapreduce job from eclipse IDE on fully distributed mode hadoop cluster?
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
I could be wrong here, but the way I understand things I do not think
that is even possible to run the JAR file from your PC. There are two
things that you need to realize.
1) How is the JAR file going to connect to the cluster
2) How is the JAR file going to be distributed to the cluster.
Again I could be wrong here in my response, so anyone else on the list
feel free to correct me. I am still a novice to Hadoop and have only
worked with it on amazon EMR.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-04-11 08:23, Answer Agrawal wrote:
> A mapreduce job can be run as jar file from terminal or directly from eclipse IDE. When a job run as jar file from terminal it uses multiple jvm and all resources of cluster. Does the same thing happen when we run from IDE. I have run a job on both and it takes less time on IDE than jar file on terminal.
>
> Thanks
Re: Can we run mapreduce job from eclipse IDE on fully distributed mode hadoop cluster?
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
I could be wrong here, but the way I understand things I do not think
that is even possible to run the JAR file from your PC. There are two
things that you need to realize.
1) How is the JAR file going to connect to the cluster
2) How is the JAR file going to be distributed to the cluster.
Again I could be wrong here in my response, so anyone else on the list
feel free to correct me. I am still a novice to Hadoop and have only
worked with it on amazon EMR.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-04-11 08:23, Answer Agrawal wrote:
> A mapreduce job can be run as jar file from terminal or directly from eclipse IDE. When a job run as jar file from terminal it uses multiple jvm and all resources of cluster. Does the same thing happen when we run from IDE. I have run a job on both and it takes less time on IDE than jar file on terminal.
>
> Thanks
Re: Can we run mapreduce job from eclipse IDE on fully distributed mode hadoop cluster?
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
I could be wrong here, but the way I understand things I do not think
that is even possible to run the JAR file from your PC. There are two
things that you need to realize.
1) How is the JAR file going to connect to the cluster
2) How is the JAR file going to be distributed to the cluster.
Again I could be wrong here in my response, so anyone else on the list
feel free to correct me. I am still a novice to Hadoop and have only
worked with it on amazon EMR.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-04-11 08:23, Answer Agrawal wrote:
> A mapreduce job can be run as jar file from terminal or directly from eclipse IDE. When a job run as jar file from terminal it uses multiple jvm and all resources of cluster. Does the same thing happen when we run from IDE. I have run a job on both and it takes less time on IDE than jar file on terminal.
>
> Thanks